Retrieve files via http

Discuss the development of new homebrew software, tools and libraries.

Moderators: cheriff, TyRaNiD

planetusa
Posts: 21
Joined: Tue Dec 06, 2005 3:03 pm
Location: Asheville
Contact:

Retrieve files via http

Post by planetusa »

Can someone provide an example of source as to the best way to retrieve files via http (port 80, and http GET) in the same basic operation as wget?

Many thanks,
Michael
mrbrown
Site Admin
Posts: 1537
Joined: Sat Jan 17, 2004 11:24 am

Post by mrbrown »

Do not use ads in your signature.
planetusa
Posts: 21
Joined: Tue Dec 06, 2005 3:03 pm
Location: Asheville
Contact:

Post by planetusa »

mrbrown wrote:Do not use ads in your signature.
Sorry bout that, I've just been copying and pasting it in lately.

As to my question, based on wifitest, I know how to open a socket for listening, and send data, but I suppose there's a cleaner more efficient way of doing it.

Michael
Shine
Posts: 728
Joined: Fri Dec 03, 2004 12:10 pm
Location: Germany

Re: Retrieve files via http

Post by Shine »

See this topic http://forums.ps2dev.org/viewtopic.php?t=4216 and my sample program. You have to strip the HTTP-headers and perhaps find a cleaner way for detecting the file end, or you have to wait until mrbrown adds the code to the PSPSDK wlan sample and cleans it up :-)
PspPet
Posts: 210
Joined: Wed Mar 30, 2005 2:13 am
Contact:

Post by PspPet »

FWIW: there is a more elaborate/robust library built into the system that does all of this. However it hasn't been completely reverse engineered (yet/ever?)
If interested, look for the "sceHttp" APIs.
Zenurb
Posts: 106
Joined: Fri Sep 30, 2005 8:33 am
Location: United Kingdom
Contact:

Post by Zenurb »

PspPet wrote:FWIW: there is a more elaborate/robust library built into the system that does all of this. However it hasn't been completely reverse engineered (yet/ever?)
If interested, look for the "sceHttp" APIs.
A full documentation of all the sce* API functions (i.e. parameter lists/usages nstuff) would be nice. Not that I know C or anything.
Proud Dvorak User
US 1.5 PSP (Original)
planetusa
Posts: 21
Joined: Tue Dec 06, 2005 3:03 pm
Location: Asheville
Contact:

Post by planetusa »

I hear tale using those same API's, there is a DNS resolve function....anybody know how to use it? Can you post in a function to return the hosts IP address?

Michael
Shine
Posts: 728
Joined: Fri Dec 03, 2004 12:10 pm
Location: Germany

Post by Shine »

planetusa wrote:I hear tale using those same API's, there is a DNS resolve function....anybody know how to use it? Can you post in a function to return the hosts IP address?
It's used in PSPRadio. I've included it in Lua Player, see luawlan.cpp and the WLAN Lua example how to use it (should be no problem to see how the Lua script calls the C functions).
planetusa
Posts: 21
Joined: Tue Dec 06, 2005 3:03 pm
Location: Asheville
Contact:

Post by planetusa »

I looked at that, and despite all my efforts, couldn't get it to work. Are you planning to update your wget code to use dns?

Michael
liquid8d
Posts: 66
Joined: Thu Jun 30, 2005 6:29 am

Post by liquid8d »

I can't get the DNS to work at all, it always hangs on ResolverStartNtoA...


liquid8d
Shine
Posts: 728
Joined: Fri Dec 03, 2004 12:10 pm
Location: Germany

Post by Shine »

I've updated it again, now it works with the latest SVN and the headers from the HTTP result are ignored. Now it is nearly ready for a new sample in the PSPSDK sample folder.

main.c:

Code: Select all

#include <string.h>
#include <errno.h>
#include <malloc.h>
#include <pspkernel.h>
#include <pspdebug.h>
#include <pspsdk.h>
#include <psputility.h>
#include <pspnet_apctl.h>
#include <pspnet_inet.h>
#include <netinet/in.h>
#include <sys/socket.h>

// TODO&#58; should be moved to PSPSDK
int sceNetResolverCreate&#40;int *rid, void *buf, SceSize buflen&#41;;
int sceNetResolverStartNtoA&#40;int rid, const char* hostname, struct in_addr* addr, unsigned int timeout, int retry&#41;;
int sceNetResolverStartAtoN&#40;int rid, const struct in_addr* addr, char *hostname, SceSize hostname_len, unsigned int timeout, int retry&#41;;
int sceNetResolverStop&#40;int rid&#41;;
int sceNetInetInetAton&#40;const char* host, struct in_addr* addr&#41;;

static char resolverBuffer&#91;1024&#93;;
static int resolverId;

/* Define the module info section */
PSP_MODULE_INFO&#40;"WGET", 0x1000, 1, 1&#41;;
PSP_MAIN_THREAD_ATTR&#40;0&#41;;
PSP_MAIN_THREAD_STACK_SIZE_KB&#40;32&#41;; /* smaller stack for kernel thread */

/* Exit callback */
int exit_callback&#40;int arg1, int arg2, void *common&#41;
&#123;
	sceKernelExitGame&#40;&#41;;
	return 0;
&#125;

/* Callback thread */
int CallbackThread&#40;SceSize args, void *argp&#41;
&#123;
	int cbid;

	cbid = sceKernelCreateCallback&#40;"Exit Callback", exit_callback, NULL&#41;;
	sceKernelRegisterExitCallback&#40;cbid&#41;;

	sceKernelSleepThreadCB&#40;&#41;;

	return 0;
&#125;

/* Sets up the callback thread and returns its thread id */
int SetupCallbacks&#40;void&#41;
&#123;
	int thid = 0;

	thid = sceKernelCreateThread&#40;"update_thread", CallbackThread, 0x11, 0xFA0, 0, 0&#41;;
	if&#40;thid >= 0&#41;
	&#123;
		sceKernelStartThread&#40;thid, 0, 0&#41;;
	&#125;

	return thid;
&#125;

void wget&#40;&#41;
&#123;
	///////////// PSP specific part ///////////////

	// init wlan
	int err = pspSdkInetInit&#40;&#41;;
	if &#40;err != 0&#41; &#123;
		pspDebugScreenPrintf&#40;"pspSdkInetInit failed&#58; %i", err&#41;;
		return;
	&#125;

	// print available connections
	pspDebugScreenPrintf&#40;"available connections&#58;\n"&#41;;
	int i;
	for &#40;i = 1; i < 100; i++&#41; // skip the 0th connection
	&#123;
		if &#40;sceUtilityCheckNetParam&#40;i&#41; != 0&#41; break;  // no more
		char name&#91;64&#93;;
		sceUtilityGetNetParam&#40;i, 0, &#40;netData*&#41; name&#41;;
		pspDebugScreenPrintf&#40;"%i&#58; %s\n", i, name&#41;;
	&#125;
	
	// use connection 1
	pspDebugScreenPrintf&#40;"using first connection\n"&#41;;
	err = sceNetApctlConnect&#40;1&#41;;
	if &#40;err != 0&#41; &#123;
		pspDebugScreenPrintf&#40;"sceNetApctlConnect failed&#58; %i", err&#41;;
		return;
	&#125;

	// WLAN is initialized when PSP IP address is available
	pspDebugScreenPrintf&#40;"init WLAN and getting IP address"&#41;;
	char pspIPAddr&#91;32&#93;;
	while &#40;1&#41; &#123;
		if &#40;sceNetApctlGetInfo&#40;8, pspIPAddr&#41; == 0&#41; break;
		pspDebugScreenPrintf&#40;"."&#41;;
		sceKernelDelayThread&#40;1000 * 1000&#41;;  // wait a second
	&#125;
	pspDebugScreenPrintf&#40;"\nPSP IP address&#58; %s\n", pspIPAddr&#41;;

	// start DNS resolver
	err = sceNetResolverCreate&#40;&resolverId, resolverBuffer, sizeof&#40;resolverBuffer&#41;&#41;;
	if &#40;err != 0&#41; &#123;
		pspDebugScreenPrintf&#40;"sceNetResolverCreate failed&#58; %i", err&#41;;
		return;
	&#125;

	// resolve host
	pspDebugScreenPrintf&#40;"resolving host...\n"&#41;;
	const char *host = "www.luaplayer.org";
	int port = 80;
	struct sockaddr_in addrTo;
	addrTo.sin_family = AF_INET;
	addrTo.sin_port = htons&#40;port&#41;;
	err = sceNetInetInetAton&#40;host, &addrTo.sin_addr&#41;;
	if &#40;err == 0&#41; &#123;
		err = sceNetResolverStartNtoA&#40;resolverId, host, &addrTo.sin_addr, 2, 3&#41;;
		if &#40;err != 0&#41; &#123;
			pspDebugScreenPrintf&#40;"sceNetResolverStartNtoA failed&#58; %i", err&#41;;
			return;
		&#125;
	&#125;


	///////////// standard socket part ///////////////

	// create socket &#40;in blocking mode&#41;
	pspDebugScreenPrintf&#40;"creating socket...\n"&#41;;
	int sock = socket&#40;AF_INET, SOCK_STREAM, 0&#41;;
	if &#40;sock < 0&#41; &#123;
		pspDebugScreenPrintf&#40;"socket failed, errno&#58; %i", errno&#41;;
		return;
	&#125;

	// connect &#40;this may block some time&#41;
	pspDebugScreenPrintf&#40;"connecting...\n"&#41;;
	err = connect&#40;sock, &#40;struct sockaddr*&#41; &addrTo, sizeof&#40;addrTo&#41;&#41;;
	if &#40;err != 0&#41; &#123;
		pspDebugScreenPrintf&#40;"connect failed"&#41;;
		return;
	&#125;

	// send HTTP request
	pspDebugScreenPrintf&#40;"loading page"&#41;;
	const char* request =
		"GET /wlan-pspsdk-test.txt HTTP/1.0\r\n"
		"host&#58; www.luaplayer.org\r\n\r\n";
	err = send&#40;sock, request, strlen&#40;request&#41;, 0&#41;;
	if &#40;err < 0&#41; &#123;
		pspDebugScreenPrintf&#40;"send failed, errno&#58; %i", errno&#41;;
		return;
	&#125;
	
	// read all
	int capacity = 256;
	int len = 0;
	u8* buffer = &#40;u8*&#41; malloc&#40;capacity&#41;;
	while &#40;1&#41; &#123;
		pspDebugScreenPrintf&#40;"."&#41;;

		// read data
		int count = recv&#40;sock, &#40;u8*&#41; &buffer&#91;len&#93;, capacity - len, 0&#41;;

		// in blocking mode it has to return something, otherwise it is closed
		// &#40;which is the default for HTTP/1.0 connections&#41;
		if &#40;count == 0&#41; break;
		
		if &#40;count < 0&#41; &#123;
			pspDebugScreenPrintf&#40;"read error&#58; %i\n", errno&#41;;
			break;
		&#125;

		// adjust buffer, if needed
		len += count;
		if &#40;len + 256 > capacity&#41; &#123;
			capacity *= 2;
			buffer = realloc&#40;buffer, capacity&#41;;
			if &#40;!buffer&#41; break;
		&#125;
	&#125;
	
	// now "len" bytes data are in buffer, add a 0 at end for printing and search for header end
	if &#40;buffer&#41; &#123;
		buffer&#91;len&#93; = 0;
		char* page = strstr&#40;&#40;char*&#41; buffer, "\r\n\r\n"&#41;;
		if &#40;page&#41; &#123;
			page += 4;
			pspDebugScreenPrintf&#40;"\n\n%s", page&#41;;
		&#125;
		free&#40;buffer&#41;;
	&#125;
	

	///////////// PSP specific part ///////////////

	// term, TODO&#58; this doesn't work
	sceNetApctlDisconnect&#40;&#41;;
	pspSdkInetTerm&#40;&#41;;
&#125;
	

int user_main&#40;SceSize argc, void* argv&#41;
&#123;
	SetupCallbacks&#40;&#41;;
	
	// test wlan
	wget&#40;&#41;;
	pspDebugScreenPrintf&#40;"\ntest finished, wlan should be deactivated now"&#41;;
	
	// wait until user ends the program
	sceKernelSleepThread&#40;&#41;;

	return 0;
&#125;

__attribute__&#40;&#40;constructor&#41;&#41; void wlanInit&#40;&#41;
&#123;
	// TODO&#58; wlan initialization should work in a kernel mode constructor
//	pspSdkLoadInetModules&#40;&#41;;
&#125;

int main&#40;void&#41;
&#123;
	pspDebugScreenInit&#40;&#41;;
	int err = pspSdkLoadInetModules&#40;&#41;;

	if &#40;err != 0&#41; &#123;
		pspDebugScreenPrintf&#40;"pspSdkLoadInetModules failed with %x\n", err&#41;;
		sceKernelDelayThread&#40;5*1000000&#41;; // 5 sec to read error
		return 1;
	&#125;

	// create user thread, tweek stack size here if necessary
	SceUID thid = sceKernelCreateThread&#40;"User Mode Thread", user_main,
		0x11, // default priority
		256 * 1024, // stack size &#40;256KB is regular default&#41;
		PSP_THREAD_ATTR_USER, NULL&#41;;
	
	// start user thread, then wait for it to do everything else
	sceKernelStartThread&#40;thid, 0, NULL&#41;;
	sceKernelWaitThreadEnd&#40;thid, NULL&#41;;
	
	sceKernelExitGame&#40;&#41;;

	return 0;

&#125;
Makefile:

Code: Select all

TARGET = wget
OBJS = main.o

INCDIR =
CFLAGS = -O2 -G0 -Wall
CXXFLAGS = $&#40;CFLAGS&#41; -fno-exceptions -fno-rtti
ASFLAGS = $&#40;CFLAGS&#41;

LIBDIR =
LDFLAGS =
LIBS = -lpspdebug -lpspsdk -lpspwlan -lpspnet_apctl -lpspnet_resolver -lc -lpspnet_inet -lpspnet
# TODO&#58; without -lpspsdk the linker reports many undefined references in pspSdkInetInit

EXTRA_TARGETS = EBOOT.PBP
PSP_EBOOT_TITLE = wget

PSPSDK=$&#40;shell psp-config --pspsdk-path&#41;
include $&#40;PSPSDK&#41;/lib/build.mak
Last edited by Shine on Sun Dec 11, 2005 8:59 am, edited 1 time in total.
Shine
Posts: 728
Joined: Fri Dec 03, 2004 12:10 pm
Location: Germany

Post by Shine »

Is anyone working on the socket support for the PSPSDK? With the latest changes in the socket files in SVN from today the program I posted doesn't even compile anymore, the linker says:
/usr/local/pspdev/lib/gcc/psp/4.0.2/../../../../psp/lib/libc.a(connect.o): In function `connect':
../../../../../../newlib/libc/sys/psp/socket.c:178: undefined reference to `sceNetInetGetErrno'
/usr/local/pspdev/lib/gcc/psp/4.0.2/../../../../psp/lib/libc.a(recv.o): In function `recv':
../../../../../../newlib/libc/sys/psp/socket.c:268: undefined reference to `sceNetInetRecv'
../../../../../../newlib/libc/sys/psp/socket.c:271: undefined reference to `sceNetInetGetErrno'
/usr/local/pspdev/lib/gcc/psp/4.0.2/../../../../psp/lib/libc.a(send.o): In function `send':
../../../../../../newlib/libc/sys/psp/socket.c:330: undefined reference to `sceNetInetSend'
../../../../../../newlib/libc/sys/psp/socket.c:333: undefined reference to `sceNetInetGetErrno'
/usr/local/pspdev/lib/gcc/psp/4.0.2/../../../../psp/lib/libc.a(socket.o): In function `socket':
../../../../../../newlib/libc/sys/psp/socket.c:43: undefined reference to `sceNetInetClose'
../../../../../../newlib/libc/sys/psp/socket.c:33: undefined reference to `sceNetInetGetErrno'
/usr/local/pspdev/lib/gcc/psp/4.0.2/../../../../psp/lib/libc.a(socket.o): In function `__psp_socket_close':
../../../../../../newlib/libc/sys/psp/socket.c:66: undefined reference to `sceNetInetClose'
../../../../../../newlib/libc/sys/psp/socket.c:70: undefined reference to `sceNetInetGetErrno'
collect2: ld returned 1 exit status
make: *** [wget.elf] Error 1
mrbrown
Site Admin
Posts: 1537
Joined: Sat Jan 17, 2004 11:24 am

Post by mrbrown »

Are you linking with libpspnet_inet.a?
Shine
Posts: 728
Joined: Fri Dec 03, 2004 12:10 pm
Location: Germany

Post by Shine »

mrbrown wrote:Are you linking with libpspnet_inet.a?
yes, see above for my LIBS definition. The compiler is called like this:

Code: Select all

psp-gcc -I. -I/usr/local/pspdev/psp/sdk/include -O2 -G0 -Wall  -L. -L/usr/local/pspdev/psp/sdk/lib   main.o -lpspdebug -lpspsdk -lpspwlan -lpspnet_apctl -lpspnet_resolver -lpspnet_inet -lpspnet -lpspdebug -lpspdisplay -lpspge -lpspctrl -lpspsdk -lc -lpsputility -lpspuser -lpspkernel -o wget.elf
But perhaps it is easier if you would just copy and past my example and fix it :-)
mrbrown
Site Admin
Posts: 1537
Joined: Sat Jan 17, 2004 11:24 am

Post by mrbrown »

-lpspnet_inet needs to be specified after -lc because the socket code in -lc depends on it.
Shine
Posts: 728
Joined: Fri Dec 03, 2004 12:10 pm
Location: Germany

Post by Shine »

mrbrown wrote:-lpspnet_inet needs to be specified after -lc because the socket code in -lc depends on it.
thanks, now the LIBS line looks like this and it works:

Code: Select all

LIBS = -lpspdebug -lpspsdk -lpspwlan -lpspnet_apctl -lpspnet_resolver -lc -lpspnet_inet -lpspnet
The last thing missing for easier porting existing socket programs would be to add the in_addr and sockaddr_in structure definitions in the psp include directory and providing the resolver stuff with the gethostbyname function (perhaps with lazy initialization of the resolver lib to avoid too many memory usage on pspSdkInetInit, if not needed and unloading it on pspSdkInetTerm, if initialized).
pspfan
Posts: 1
Joined: Fri Dec 30, 2005 4:59 am

missing socket.h

Post by pspfan »

Hello,
I installed the latest pspdev using both cygwin (+ pspchain) and devkitPro but I got this error when I try to compile this example :

psp-gcc -I. -I/usr/local/pspdev/psp/sdk/include -O2 -G0 -Wall -c -o main.o mai
n.c
main.c:10:25: error: netinet/in.h: No such file or directory
main.c:11:25: error: sys/socket.h: No such file or directory
main.c:15: warning: 'struct in_addr' declared inside parameter list
main.c:15: warning: its scope is only this definition or declaration, which is p
robably not what you want
main.c:16: warning: 'struct in_addr' declared inside parameter list
main.c:18: warning: 'struct in_addr' declared inside parameter list
main.c: In function 'wget':
main.c:113: error: storage size of 'addrTo' isn't known
main.c:114: error: 'AF_INET' undeclared (first use in this function)
main.c:114: error: (Each undeclared identifier is reported only once
main.c:114: error: for each function it appears in.)
main.c:115: warning: implicit declaration of function 'htons'
main.c:130: warning: implicit declaration of function 'socket'
main.c:130: error: 'SOCK_STREAM' undeclared (first use in this function)
main.c:138: warning: implicit declaration of function 'connect'
main.c:149: warning: implicit declaration of function 'send'
main.c:163: warning: implicit declaration of function 'recv'
main.c:113: warning: unused variable 'addrTo'
make: *** [main.o] Error 1

I suppose that the problem lies in an incorrect library path when compiling the example (it doesn't found sys/socket.h and other include files)

the only file I found (under cygwin installation that seems the missing ones are under the /home/local/include but when I include those files I got another error).

Thanks in advance for the help.

Francesco.
planetusa
Posts: 21
Joined: Tue Dec 06, 2005 3:03 pm
Location: Asheville
Contact:

Post by planetusa »

I have modified Shine's example heavily (sent him a PM), it now supports URI parsing, and both the GET and POST methods, and I will be happy to share the code, as soon as I can get binary files to transfer correctly. Currently, plaintext works flawlessly, if you can help, please see thread

http://forums.ps2dev.org/viewtopic.php? ... highlight=

This code can be used for many many things, the possibilities are endless.


Thanks,
Michael
planetusa
Posts: 21
Joined: Tue Dec 06, 2005 3:03 pm
Location: Asheville
Contact:

Post by planetusa »

Code: Select all

#include <string.h> 
#include <stdio.h>
#include <errno.h> 
#include <malloc.h> 
#include <pspkernel.h> 
#include <pspdebug.h> 
#include <pspsdk.h> 
#include <psputility.h> 
#include <pspnet_apctl.h> 
#include <pspnet_inet.h> 
#include <netinet/in.h> 
#include <sys/socket.h> 
#include <util.h>

// TODO&#58; should be moved to PSPSDK 
int sceNetResolverCreate&#40;int *rid, void *buf, SceSize buflen&#41;; 
int sceNetResolverStartNtoA&#40;int rid, const char* hostname, struct in_addr* addr, unsigned int timeout, int retry&#41;; 
int sceNetResolverStartAtoN&#40;int rid, const struct in_addr* addr, char *hostname, SceSize hostname_len, unsigned int timeout, int retry&#41;; 
int sceNetResolverStop&#40;int rid&#41;; 
int sceNetInetInetAton&#40;const char* host, struct in_addr* addr&#41;; 
#define SOCKET int
#define MAX_URL_LENGTH 750
#define MAX_JPEG	30000
typedef struct &#123;
	char uri&#91;256&#93;;
    char sheme&#91;256&#93;;
    char host&#91;256&#93;;
    char abs_path&#91;256&#93;;
    int port;
    char query&#91;256&#93;;
&#125; URI;

static char resolverBuffer&#91;1024&#93;; 
static int resolverId; 

/* Define the module info section */ 
PSP_MODULE_INFO&#40;"WGET", 0x1000, 1, 1&#41;; 
PSP_MAIN_THREAD_ATTR&#40;0&#41;; 
PSP_MAIN_THREAD_STACK_SIZE_KB&#40;32&#41;; /* smaller stack for kernel thread */ 

/* Exit callback */ 
int exit_callback&#40;int arg1, int arg2, void *common&#41; 
&#123; 
   sceKernelExitGame&#40;&#41;; 
   return 0; 
&#125; 

/* Callback thread */ 
int CallbackThread&#40;SceSize args, void *argp&#41; 
&#123; 
   int cbid; 

   cbid = sceKernelCreateCallback&#40;"Exit Callback", exit_callback, NULL&#41;; 
   sceKernelRegisterExitCallback&#40;cbid&#41;; 

   sceKernelSleepThreadCB&#40;&#41;; 

   return 0; 
&#125; 

/* Sets up the callback thread and returns its thread id */ 
int SetupCallbacks&#40;void&#41; 
&#123; 
   int thid = 0; 

   thid = sceKernelCreateThread&#40;"update_thread", CallbackThread, 0x11, 0xFA0, 0, 0&#41;; 
   if&#40;thid >= 0&#41; 
   &#123; 
      sceKernelStartThread&#40;thid, 0, 0&#41;; 
   &#125; 

   return thid; 
&#125; 

unsigned short htons&#40;unsigned short wIn&#41;
&#123;
    u8 bHi = &#40;wIn >> 8&#41; & 0xFF;
    u8 bLo = wIn & 0xFF;
    return &#40;&#40;unsigned short&#41;bLo << 8&#41; | bHi;
&#125;


void ParseURI&#40;URI *uri&#41;
&#123;
 /* parsing address

      http_URL = "http&#58;" "//" host &#91; "&#58;" port &#93; &#91; abs_path &#91; "?" query &#93;&#93;

    */

    *uri->sheme=0;
 	*uri->host=0;
    strcpy&#40;uri->abs_path,"/"&#41;;
    *uri->query=0;
    uri->port=0;

	char *start;
    char *end;
    char portstr&#91;256&#93;;


    start=uri->uri;
	//end=strstr&#40;start,"&#58;//"&#41;;
    end=strchr&#40;start,'&#58;'&#41;;

    if &#40; end != NULL&#41; &#123;
    	char *end1=strstr&#40;end,"//"&#41;;
    	if &#40;end1 != NULL&#41; &#123;
        	if &#40;end < end1 &#41; &#123;
    			strncpy&#40;uri->sheme,start,end-start+1&#41;;
        		uri->sheme&#91;end-start+1&#93;=0;
        		strlwr&#40;uri->sheme&#41;;
            &#125;
            start=end1+2;
        &#125;
        else &#123;

        	int num=atoi&#40;end+1&#41;;
            if &#40; num <= 0 &#41; &#123;
         		strncpy&#40;uri->sheme,start,end-start+1&#41;;
            	uri->sheme&#91;end-start+1&#93;=0;
            	strlwr&#40;uri->sheme&#41;;
            	start=end+1;
            &#125;
        &#125;
    &#125;

    end=uri->uri+strlen&#40;uri->uri&#41;;
    char *slash=strchr&#40;start,'/'&#41;;
    char *colon=strchr&#40;start,'&#58;'&#41;;
    char *qmark=strchr&#40;start,'\?'&#41;;

    if &#40;qmark != NULL &#41; &#123; // query
    	strcpy&#40;uri->query,qmark&#41;;
        end=qmark;
    &#125;


    if &#40;slash != NULL &#41; &#123; // abs_path
    	if &#40;slash < end&#41; &#123;
    		strncpy&#40;uri->abs_path,slash,end-slash&#41;;
            uri->abs_path&#91;end-slash&#93;=0;
            end=slash;
        &#125;
    &#125;


    if &#40; colon != NULL && colon < end &#41; &#123;
    	strncpy&#40;portstr,colon+1,end-colon-1&#41;;
        portstr&#91;end-colon-1&#93;=0;
        end=colon;
        uri->port=atoi&#40;portstr&#41;;
    &#125;

    if &#40;end > start&#41; &#123;
    	strncpy&#40;uri->host,start,end-start&#41;;
        uri->host&#91;end-start&#93;=0;
        strlwr&#40;uri->host&#41;;

    &#125;
&#125;




void wget&#40;URI *uri, char* method, char* formdata&#41; 
&#123; 
   ///////////// PSP specific part /////////////// 
	pspDebugScreenPrintf&#40;"downloading %s now\n", uri&#41;;
   // init wlan 
   int err = pspSdkInetInit&#40;&#41;; 
   if &#40;err != 0&#41; &#123; 
      pspDebugScreenPrintf&#40;"pspSdkInetInit failed&#58; %i", err&#41;; 
      return; 
   &#125; 
	//init_file&#40;"ms0&#58;/custom.html"&#41;;
   // print available connections 
   //pspDebugScreenPrintf&#40;"available connections&#58;\n"&#41;; //removed
   ParseURI&#40;uri&#41;;
   
   int i; 
   for &#40;i = 1; i < 100; i++&#41; // skip the 0th connection 
   &#123; 
      if &#40;sceUtilityCheckNetParam&#40;i&#41; != 0&#41; break;  // no more 
      char name&#91;64&#93;; 
      sceUtilityGetNetParam&#40;i, 0, &#40;netData*&#41; name&#41;; 
      pspDebugScreenPrintf&#40;"%i&#58; %s\n", i, name&#41;; 
   &#125; 
    
   // use connection 1 
   //pspDebugScreenPrintf&#40;"using first connection\n"&#41;; //removed
   err = sceNetApctlConnect&#40;1&#41;; 
   if &#40;err != 0&#41; &#123; 
      pspDebugScreenPrintf&#40;"sceNetApctlConnect failed&#58; %i", err&#41;; 
      return; 
   &#125; 

   // WLAN is initialized when PSP IP address is available 
   //pspDebugScreenPrintf&#40;"init WLAN and getting IP address"&#41;; //removed
   char pspIPAddr&#91;32&#93;; 
   while &#40;1&#41; &#123; 
      if &#40;sceNetApctlGetInfo&#40;8, pspIPAddr&#41; == 0&#41; break; 
      pspDebugScreenPrintf&#40;"."&#41;; 
      sceKernelDelayThread&#40;1000 * 1000&#41;;  // wait a second 
   &#125; 
   //pspDebugScreenPrintf&#40;"\nPSP IP address&#58; %s\n", pspIPAddr&#41;; //removed

   // start DNS resolver 
   err = sceNetResolverCreate&#40;&resolverId, resolverBuffer, sizeof&#40;resolverBuffer&#41;&#41;; 
   if &#40;err != 0&#41; &#123; 
      pspDebugScreenPrintf&#40;"sceNetResolverCreate failed&#58; %i", err&#41;; 
      return; 
   &#125; 

   // resolve host 
   //pspDebugScreenPrintf&#40;"resolving host...\n"&#41;; //removed
   const char *host = uri->host; 
   int port = 80; 
   struct sockaddr_in addrTo; 
   addrTo.sin_family = AF_INET; 
   addrTo.sin_port = htons&#40;port&#41;; 
   err = sceNetInetInetAton&#40;host, &addrTo.sin_addr&#41;; 
   if &#40;err == 0&#41; &#123; 
      err = sceNetResolverStartNtoA&#40;resolverId, host, &addrTo.sin_addr, 2, 3&#41;; 
      if &#40;err != 0&#41; &#123; 
         pspDebugScreenPrintf&#40;"sceNetResolverStartNtoA failed&#58; %i", err&#41;; 
         return; 
      &#125; 
   &#125; 


   ///////////// standard socket part /////////////// 

   // create socket &#40;in blocking mode&#41; 
   //pspDebugScreenPrintf&#40;"creating socket...\n"&#41;;  //removed
   int sock = socket&#40;AF_INET, SOCK_STREAM, 0&#41;; 
   if &#40;sock < 0&#41; &#123; 
      pspDebugScreenPrintf&#40;"socket failed, errno&#58; %i", errno&#41;; 
      return; 
   &#125; 

   // connect &#40;this may block some time&#41; 
   //pspDebugScreenPrintf&#40;"connecting...\n"&#41;;  //removed
   err = connect&#40;sock, &#40;struct sockaddr*&#41; &addrTo, sizeof&#40;addrTo&#41;&#41;; 
   if &#40;err != 0&#41; &#123; 
      pspDebugScreenPrintf&#40;"connect failed"&#41;; 
      return; 
   &#125; 

   // send HTTP request 
   //pspDebugScreenPrintf&#40;"loading page"&#41;;  //removed
   
   //const char* request = 
   //"GET / HTTP/1.0\r\n"
   //"host&#58; www.google.com\r\n\r\n";
   const char request&#91;MAX_URL_LENGTH + 1&#93; = &#123; '\0' &#125;;
   if &#40;method == "GET"&#41; &#123;
		sprintf&#40;request, 
		"GET %s HTTP/1.0\r\n"
		"host&#58; %s\r\n\r\n", 
		uri->abs_path, uri->host&#41;;
   &#125;   
	if &#40;method == "POST"&#41; &#123;
		sprintf&#40;request, 
		"POST %s HTTP/1.0\r\nHost&#58; %s\r\nUser-Agent&#58; PSP\r\nContent-type&#58; application/x-www-form-urlencoded\r\nContent-length&#58; %u\r\n\r\n%s",
		uri->abs_path, uri->host, strlen&#40;formdata&#41;, formdata&#41;;
	&#125;
	pspDebugScreenPrintf&#40;request&#41;; //for testing
   err = send&#40;sock, request, strlen&#40;request&#41;, 0&#41;; 
   if &#40;err < 0&#41; &#123; 
      pspDebugScreenPrintf&#40;"send failed, errno&#58; %i", errno&#41;; 
      return; 
   &#125; 
   int capacity = 256; 
   int len = 0; 
   u8* buffer = &#40;u8*&#41; malloc&#40;capacity&#41;; 
   
   while &#40;1&#41; &#123; 
      pspDebugScreenPrintf&#40;"."&#41;; 

      // read data 
      int count = recv&#40;sock, &#40;u8*&#41; &buffer&#91;len&#93;, capacity - len, 0&#41;; 

      // in blocking mode it has to return something, otherwise it is closed 
      // &#40;which is the default for HTTP/1.0 connections&#41; 
      if &#40;count == 0&#41; break; 
       
      if &#40;count < 0&#41; &#123; 
         pspDebugScreenPrintf&#40;"read error&#58; %i\n", errno&#41;; 
         break; 
      &#125; 

      // adjust buffer, if needed 
      len += count; 
      if &#40;len + 256 > capacity&#41; &#123; 
         capacity *= 2;
         buffer = realloc&#40;buffer, capacity&#41;; 
         if &#40;!buffer&#41; break; 
      &#125; 
   &#125; 
    // now "len" bytes data are in buffer, add a 0 at end for printing and search for header end 
   //pspDebugScreenPrintf&#40;buffer&#41;; //used for debugging
   if &#40;buffer&#41; &#123; 
      buffer&#91;len&#93; = 0; 
      char* page = strstr&#40;&#40;char*&#41; buffer, "\r\n\r\n"&#41;; 
      if &#40;page&#41; &#123; 
         page += 4; 
         //pspDebugScreenPrintf&#40;page&#41;; //only for testing output
		 //parse_html&#40;page&#41;; 
		 //write_to_file&#40;"ms0&#58;/custom.html", page&#41;;
		 pspDebugScreenPrintf&#40;"\n\nPage saved to ms0&#58;/custom.html"&#41;; 
      &#125; 
      free&#40;buffer&#41;; 
   &#125; 
    

   ///////////// PSP specific part /////////////// 

   // term, TODO&#58; this doesn't work 
   sceNetApctlDisconnect&#40;&#41;; 
   pspSdkInetTerm&#40;&#41;; 
&#125; 
    

int user_main&#40;SceSize argc, void* argv&#41; 
&#123; 
   SetupCallbacks&#40;&#41;; 
    
   // test wlan 
   // wget&#40;URL, HTTP METHOD, FORM DATA&#41; POST for POST GET for GET
   wget&#40;"www.google.com","GET","field1=this&field2=is&field3=nota&field4=test"&#41;; // Any plaintext url should work
   //pspDebugScreenPrintf&#40;"\ntest finished, wlan should be deactivated now"&#41;; //removed
   
   // wait until user ends the program 
   sceKernelSleepThread&#40;&#41;; 

   return 0; 
&#125; 

__attribute__&#40;&#40;constructor&#41;&#41; void wlanInit&#40;&#41; 
&#123; 
   // TODO&#58; wlan initialization should work in a kernel mode constructor 
//   pspSdkLoadInetModules&#40;&#41;; 
&#125; 

int main&#40;void&#41; 
&#123; 
   pspDebugScreenInit&#40;&#41;; 
   int err = pspSdkLoadInetModules&#40;&#41;; 

   if &#40;err != 0&#41; &#123; 
      pspDebugScreenPrintf&#40;"pspSdkLoadInetModules failed with %x\n", err&#41;; 
      sceKernelDelayThread&#40;5*1000000&#41;; // 5 sec to read error 
      return 1; 
   &#125; 

   // create user thread, tweek stack size here if necessary 
   SceUID thid = sceKernelCreateThread&#40;"User Mode Thread", user_main, 
      0x11, // default priority 
      256 * 1024, // stack size &#40;256KB is regular default&#41; 
      PSP_THREAD_ATTR_USER, NULL&#41;; 
    
   // start user thread, then wait for it to do everything else 
   sceKernelStartThread&#40;thid, 0, NULL&#41;; 
   sceKernelWaitThreadEnd&#40;thid, NULL&#41;; 
    
   sceKernelExitGame&#40;&#41;; 

   return 0; 

&#125;

It should compile without issue, I commented out functions that are specific to what I'm doing. If anyone gets binary files to work, jpg, etc, please let me know.

Michael
dbeyer3069
Posts: 81
Joined: Mon Dec 19, 2005 4:09 pm

Post by dbeyer3069 »

Can you drop your makefile also just for giggles?

Thanks,

David Beyer
planetusa
Posts: 21
Joined: Tue Dec 06, 2005 3:03 pm
Location: Asheville
Contact:

Post by planetusa »

Code: Select all

TARGET = wget
OBJS = main.o

INCDIR = 
CFLAGS = -O2 -G0 -Wall
CXXFLAGS = $&#40;CFLAGS&#41; -fno-exceptions -fno-rtti
ASFLAGS = $&#40;CFLAGS&#41;

LIBDIR =
LDFLAGS =
LIBS = -lpspdebug -lpspsdk -lpspwlan -lpspnet_apctl -lpspnet_resolver -lc -lpspnet_inet -lpspnet
#LIBS = -lpspdebug -lpspwlan -lpspsdk -lpspnet_apctl -lpspnet_resolver -lpspnet_inet -lpspnet
# TODO&#58; without -lpspsdk it says&#58; libpspsdk.a&#40;inethelper.o&#41;&#58; In function `pspSdkInetInit'&#58; inethelper.c&#58;67&#58; undefined reference to `sceNetInit'

EXTRA_TARGETS = EBOOT.PBP
PSP_EBOOT_TITLE = wget

PSPSDK=$&#40;shell psp-config --pspsdk-path&#41;
include $&#40;PSPSDK&#41;/lib/build.mak
digihoe
Posts: 108
Joined: Sat May 14, 2005 7:40 pm

Post by digihoe »

Great to see you getting closer to file support, will be nice to see this work...

Good luck with the web browser!

Best regards!
danzel
Posts: 182
Joined: Fri Nov 04, 2005 11:03 pm

Post by danzel »

Wouldn't it be much nicer to use libcurl ;)

Here, I've just finished porting it.

http://localhost.geek.nz/crap/curl-psp.tar.bz2
Edit: Woops, its actually a .bz2 file *renamed*

Instructions:
Download the latest curl source code (I used 7.15.1).

Apply the patch and build curl: (change ../patch-curl to the location of the patch)

Code: Select all

tar -jxf curl-7.15.1.tar.bz2
cd curl-7.15.1
cat ../curl-psp-patch | patch -p1
chmod +x buildit
./buildit
(buildit is a short script to create some empty folders needed and also contains the huge configure line)

now you should have libcurl.a inside curl-7.15.1/lib/.libs/

To use libcurl you will need to link against libpsp_req.a from inside the psp_req folder and libcurl.a
(psp_req is a quick implementation of gethostbyname)

Copy both of these to the example directory and compile it with make.
(You will probally need to change the -I../curl-7.15.1 line in the Makefile)
The included example downloads my website and throws all the html on the screen :)

I haven't tryed to build an ssl library to get https support, but it will likely work, I also haven't got zlib installed so I dont know if that works ;)

Oh, The example seems to hang when you try to quit, not sure why... might be the resolver created in gethostbyname.
Last edited by danzel on Fri Jan 20, 2006 7:47 am, edited 1 time in total.
dbeyer3069
Posts: 81
Joined: Mon Dec 19, 2005 4:09 pm

Post by dbeyer3069 »

danzel wrote:Wouldn't it be much nicer to use libcurl ;)

Here, I've just finished porting it.

http://localhost.geek.nz/crap/curl-psp.tar.gz

Instructions:
Download the latest curl source code (I used 7.15.1).

Apply the patch and build curl: (change ../patch-curl to the location of the patch)

Code: Select all

tar -jxf curl-7.15.1.tar.bz2
cd curl-7.15.1
cat ../curl-psp-patch | patch -p1
chmod +x buildit
./buildit
(buildit is a short script to create some empty folders needed and also contains the huge configure line)

now you should have libcurl.a inside curl-7.15.1/lib/.libs/

To use libcurl you will need to link against libpsp_req.a from inside the psp_req folder and libcurl.a
(psp_req is a quick implementation of gethostbyname)

Copy both of these to the example directory and compile it with make.
(You will probally need to change the -I../curl-7.15.1 line in the Makefile)
The included example downloads my website and throws all the html on the screen :)

I haven't tryed to build an ssl library to get https support, but it will likely work, I also haven't got zlib installed so I dont know if that works ;)

Oh, The example seems to hang when you try to quit, not sure why... might be the resolver created in gethostbyname.
Did you get binary transfers working with this?

David Beyer
planetusa
Posts: 21
Joined: Tue Dec 06, 2005 3:03 pm
Location: Asheville
Contact:

Post by planetusa »

Well ok then....I'll give curl a try. See, we were trying to keep it light weight, but I'll take a look at what you've got.

If you've got the already patched version, if would be appreciated if you could email it to me)

Michael
planetusa
Posts: 21
Joined: Tue Dec 06, 2005 3:03 pm
Location: Asheville
Contact:

Post by planetusa »

If curl works, this makes my job a hell of a lot easier.

Michael
danzel
Posts: 182
Joined: Fri Nov 04, 2005 11:03 pm

Post by danzel »

I see no reason that binary transfers wouldnt work with this, but I haven't tested it. (peldet can do binary transfers fine, so its not a psp issue)

I've uploaded a prebuilt version here: http://localhost.geek.nz/crap/curl_psp_prebuilt.tar.bz2
The example is included, and should build straight off.

Another lib you may want (although it may be a bit big) is libxml2 http://xmlsoft.org

You can build it (atleast version 2.6.22) with the following lines:

Code: Select all

CC=psp-gcc CXX=psp-g++ LIBS="-lpspdebug -lpspdisplay -lpspge -lc -lpspuser -lpspkernel" LDFLAGS="-L/usr/local/pspdev/psp/sdk/lib" ./configure --host=mips --with-minimum --without-threads --without-writer --without-modules
make
It errors out partway through when building the tests, however this is after it successfully builds the library in the .libs dir.
I haven't played with this yet, just tryed to build it to see if it would. This is also probally not the most minimal build, you could definately make it smaller.

There is a libcurl + libxml example here: http://curl.haxx.se/lxr/source/docs/exa ... mltitle.cc
planetusa
Posts: 21
Joined: Tue Dec 06, 2005 3:03 pm
Location: Asheville
Contact:

Post by planetusa »

It has an issue with select.h, can you put that up?

Michael
danzel
Posts: 182
Joined: Fri Nov 04, 2005 11:03 pm

Post by danzel »

<sys/select.h> is part of the pspsdk, make sure you have the latest version and the latest newlib patch, as there has been a few changes in there to do with the networking stuff lately.

Code: Select all

svn checkout svn&#58;//svn.pspdev.org/psp/trunk/psptoolchain
cd psptoolchain
./toolchain.sh
copy the .tar.bz2/gz of the newlib/gcc/binutils/gdb source files into the psptoolchain folder before running ./toolchain.sh if you want to save on downloading them again.

Edit: Duh, figured out why the sample wont let you quit, its because of this:

Code: Select all

/* Exit callback */
int exit_callback&#40;int arg1, int arg2, void *common&#41;
&#123;
+++    sceKernelExitGame&#40;&#41;;
    return 0;
&#125;
There was no sceKernelExitGame(); in there. Add it and you can quit.
planetusa
Posts: 21
Joined: Tue Dec 06, 2005 3:03 pm
Location: Asheville
Contact:

Post by planetusa »

Updated my toolchain, that fixed it.....now....it downloads binary files, but my old method for writing to the MS causes the PSP to crash....so I'm looking for a new way to store the data it downloads. I've tried a few different things in place of your "spamming the screen", but so far, they either crash the psp, or don't work. Recommendations to save the returned data to a buffer so we can read it into libjpeg, or our parser, or such?

Michael
Post Reply