PS3SDK collaboration invitation

Technical discussion on the newly released and hard to find PS3.

Moderators: cheriff, emoon

User avatar
mc
Posts: 211
Joined: Wed Jan 12, 2005 7:32 am
Location: Linköping

PS3SDK collaboration invitation

Post by mc »

Well, we've all heard the rumors about ongoing PS3SDK development efforts,
but so far nobody has shown any code.

I think the main problem with getting a PS3SDK out the door is that
it's a rather large project for one or two persons to undertake, so eventually
anyone who tries on his own will get stuck and/or fed up.

So what I'd like to do is invite to a collaborative effort, and the correct way to
do this is of course to present all the code I have so far, and allow people
to start expanding it.

What I have currently is a small multithreading kernel (supports use of both
HW threads), integration with newlib, and some simple GPU support.

There are plenty of work packages for interrested parties to get down with,
including (but not limited to)

* Interrupt handling
* A more sophisticated MMU driver
* Networking
* USB (the driver in mcload can't be used as it contains GPL code)
* Storage drivers and filesystems
* SPE support
* VMX support (currently context switching does not preserve vector registers)

To get going, check out the following bzr repository:

http://mc.pp.se/bzr/ps3sdk

Note that you will need to rebuild newlib with the patch contained in
the "misc" directory (simply copy the patch over the old one in the ps3toolchain
distribution).

If you want to start working on something, please let me know, and preferably
publish the URL of your own bzr branch, so that others can follow your progress,
and so that I can merge things back to my main branch.

Let's get hacking, people!
Flying at a high speed
Having the courage
Getting over crisis
I rescue the people
User avatar
jbit
Site Admin
Posts: 293
Joined: Sat May 28, 2005 3:11 am
Location: København, Danmark
Contact:

Post by jbit »

It would be nice if we can keep the core GPL clean, as in all code is BSD/AFPL/whatever licensed, which means of course no copying code/headers/etc from linux..
GPL causes a lot of headaches at later stages for a lot of people, so it's best to avoid for this kind of thing.

(Other peoples opinions are welcome of course :P)
User avatar
mc
Posts: 211
Joined: Wed Jan 12, 2005 7:32 am
Location: Linköping

Post by mc »

@jbit: Yes, that's why I mentioned this issue with the existing USB driver.

I've reused the LICENSE from the PSPSDK.

Currently I have lv1callarg.h, lib1calltab.h and hv.S still copied from Linux for generating
the HV call stubs, so I guess another work package would be to cleanroom that.
Flying at a high speed
Having the courage
Getting over crisis
I rescue the people
User avatar
jbit
Site Admin
Posts: 293
Joined: Sat May 28, 2005 3:11 am
Location: København, Danmark
Contact:

Post by jbit »

Ah, I missed that sorry... But even so it makes sense to make people aware :)

Also I think we have enough info on the wiki about the lv1 interface to validly make a cleanroom BSD implementation. Should only be a days work or so.
User avatar
mc
Posts: 211
Joined: Wed Jan 12, 2005 7:32 am
Location: Linköping

Post by mc »

Yup, then information in the wiki is more than adequate (you actually only need
<name, id, # of inputs, # of outputs> for each call). I've already started creating
a header file with prototypes for the HV calls based on the wiki (src/hypervisor/ps3hv.h).

Hm, one file I'm a little unsure about is the linkfile, which is based on the linker
script generated by binutils. It does not contain any license information,
so I don't know if it is GPL, LGPL, or something different.
Flying at a high speed
Having the courage
Getting over crisis
I rescue the people
J.F.
Posts: 2906
Joined: Sun Feb 22, 2004 11:41 am

Post by J.F. »

It would also be helpful if the different parts (like usb or net) were separate and independent as much as possible. For example, let's say you made a PC or Mac emulator and just wanted to pass the hardware for the USB on to the emulator. If you FORCE people to load a USB driver, then you also FORCE people to emulate the hardware using the driver in some cases, like the one just mentioned. Now clearly an input driver would require a USB driver, but if you don't use anything that needs it, don't force people to load/use it anyway.
User avatar
mc
Posts: 211
Joined: Wed Jan 12, 2005 7:32 am
Location: Linköping

Post by mc »

The approach so far is pretty modular. I don't even FORCE anyone to turn on the MMU,
although of course most drivers won't work without it... ;-)
Flying at a high speed
Having the courage
Getting over crisis
I rescue the people
J.F.
Posts: 2906
Joined: Sun Feb 22, 2004 11:41 am

Post by J.F. »

mc wrote:The approach so far is pretty modular. I don't even FORCE anyone to turn on the MMU,
although of course most drivers won't work without it... ;-)
Good. I've done a checkout and will look it over to see what I can contribute.
ps2devman
Posts: 259
Joined: Mon Oct 09, 2006 3:56 pm

Post by ps2devman »

Strong advise : implement right from the beginning a strong 'DPC' mechanism for all interrupts. If you get strange and impossible to understand issues think about interrupt that fires while you are already handling the previous one. The direct interrupt handling routing must allow reentrance and be as fast as possible. Don't treat interrupt there, just enqueue what you need to treat it later. Then have something that dequeues sequentially events and call clean DPC's (deferred procedure calls) to treat them. The DPC's are normal routines and can be interrupted anytime like any other code part.

(can someone write the url to some page that explain how bzr work?
for me page is empty...)
User avatar
mc
Posts: 211
Joined: Wed Jan 12, 2005 7:32 am
Location: Linköping

Post by mc »

Yes, I was also thinking about this kind of solution for interrupt processing. I have experience from work with commercial microkernels which use this method, and it really makes stuff easier for the clients. For it to work properly, I think preemption needs to be added to the scheduler though (it doesn't have it yet).

For a quick bzr tutorial, see http://doc.bazaar-vcs.org/latest/en/min ... index.html. Basically, you just do "bzr branch htpp://mc.pp.se/bzr/ps3dev" to get your own branch that you can commit locally to.
Flying at a high speed
Having the courage
Getting over crisis
I rescue the people
nbauernf
Posts: 11
Joined: Tue Nov 18, 2008 11:35 am
Location: Carnegie Mellon University

Post by nbauernf »

Hmm I'm having trouble checking the branch out.

$ bzr branch http://mc.pp.se/bzr/ps3sdk
bzr: ERROR: Invalid http response for http://mc.pp.se/bzr/ps3sdk/: Unable to handle http code 501: Not Implemented

Any clues?
nbauernf
Posts: 11
Joined: Tue Nov 18, 2008 11:35 am
Location: Carnegie Mellon University

Post by nbauernf »

Turns out adding a '--no-plugins' took care of that problem!
gkr
Posts: 1
Joined: Thu Jul 09, 2009 5:19 am

Post by gkr »

Hi,
I get error message when trying to check out

c:\Python24\Scripts>bzr branch http://mc.pp.se/bzr/ps3sdk
bzr: ERROR: exceptions.NotImplementedError: We must have one of fcntl, pywin32,
or ctypes available to support OS locking.
User avatar
mc
Posts: 211
Joined: Wed Jan 12, 2005 7:32 am
Location: Linköping

Post by mc »

@gkr: Did you follow the instructions here?

Or maybe you are using MS-DOS?
Flying at a high speed
Having the courage
Getting over crisis
I rescue the people
ps2devman
Posts: 259
Joined: Mon Oct 09, 2006 3:56 pm

Post by ps2devman »

If you install the windows version of bazaar with minimum installation
(no plugin) it definitely works without a fuss (at least on xp)

bazaar in 5 minutes :
http://doc.bazaar-vcs.org/bzr.dev/en/mi ... index.html

You don't even need bazaar in the path if you specify a path for bzr in a .bat file. Create a .bat file in a c:\bzr_projects directory with this inside :

c:\progra~1\bazaar\bzr whoami "myname <myemail>"
c:\progra~1\bazaar\bzr branch http://mc.pp.se/bzr/ps3sdk

and launch it
rapso
Posts: 140
Joined: Mon Mar 28, 2005 6:35 am

Post by rapso »

A homebrew PS3SDK would be definitely incredibily cool :),
but from the text here it rather seems to be another OS. is there something I miss? (that's no offense. I just don't really see the goal of this project as the topic seems to promise something else than the content, imho).


rapso
J.F.
Posts: 2906
Joined: Sun Feb 22, 2004 11:41 am

Post by J.F. »

OTHEROS means the program is running on bare metal with only the hypervisor below it. There are no libraries or anything else to help do stuff, so yes, it should seem like another OS when discussing the SDK - it needs to do everything the OS would do.
rapso
Posts: 140
Joined: Mon Mar 28, 2005 6:35 am

Post by rapso »

so it's mcload with BSD license?

for me, it wouldn't need to do all the stuff to get into development. flat memory (instead of an sophisticated MMU driver), no context switching (just a call into main(..) and main2(...) with two HW threads), video out and pad would be all needed from an OS to get into it, similar to PS2 dev.

I don't say this is bad, I even praise you guys for doing such a community work, I just want to say I (and maybe other guys) would be happy about far less, as long as it's simple to get running and with access to basic hardware to get (homebrew)games running.
User avatar
emoon
Posts: 91
Joined: Sun Jan 18, 2004 10:03 pm
Location: Stockholm, Sweden
Contact:

Post by emoon »

I will release the code I have worked on to mc.

It has a semi working Ethernet driver that can actually receive and start an elf (the code is very crude and needs lots of work though)

I don't have much time to work on it but hopefully it will give some base to work from at least.
User avatar
emoon
Posts: 91
Joined: Sun Jan 18, 2004 10:03 pm
Location: Stockholm, Sweden
Contact:

Post by emoon »

Ok, I actually forgot stuff I had implemented :)

* tcp/ip (semi working) and is using LWIP
* More cleaned up code
* echo server over tcp/ip (telenet style that echos back what you send it)
* lv1 calls renamed to lv1NetControl(...) instead of lv1_net_control(...) to try to remove the 'GPLisnes' of it and also some other changes to try to make it different from the linux code
* Totally custom written smb support (to be used to load files remote) It has been implemented by Chip and is way more lean than samba and focus on whats needed (never tested on ps3 afaik but code is clean and documented)
* Includes docs on the coding standard we intended to use
* Includes a early version of ps3link (its able to kick elfs, but not very tested and needs to be fixed up to be really useable)
* Also a early version of ps3client is included.

Hopefully you will find the work that I and Chip have done useful
User avatar
mc
Posts: 211
Joined: Wed Jan 12, 2005 7:32 am
Location: Linköping

Post by mc »

emoon wrote:* lv1 calls renamed to lv1NetControl(...) instead of lv1_net_control(...) to try to remove the 'GPLisnes' of it
Actually, I'm not sure this is productive. You can't copyright an API, and I'm pretty sure these names come from the IBM/Sony HV code anyway, so I don't see any "GPLisnes" to remove as far as the actual names are concerned.

But if others have a different opinion, feel free to motivate. :)
Flying at a high speed
Having the courage
Getting over crisis
I rescue the people
User avatar
mc
Posts: 211
Joined: Wed Jan 12, 2005 7:32 am
Location: Linköping

Post by mc »

@rapso:

If you don't want to use the MMU driver, that's fine. Just don't call ps3MmuInit () and ps3MmuOn(), and you'll get your flat memory model. Note however that due to the way hardware virtualization works in the Cell, the flat memory model contains 128MB of RAM _only_. No video memory, no USB register file, no nothing. So for things like the console driver and any upcoming USB driver, the fancy schmancy MMU stuff is _required_. It isn't there just for kicks.

Same with the scheduler; if you just create one new thread from main (your main2), and never call any functions which can deschedule (sleep, mutex operations etc), you will never get any context switches. But some divers may require them because it is impractical to make them otherwise,
Flying at a high speed
Having the courage
Getting over crisis
I rescue the people
rapso
Posts: 140
Joined: Mon Mar 28, 2005 6:35 am

Post by rapso »

thx for clarifying that MC, I wasn't aware of all those implications that makes all those tasks u listed a _must_ (I really miss the console feeling tho)
J.F.
Posts: 2906
Joined: Sun Feb 22, 2004 11:41 am

Post by J.F. »

rapso wrote:(I really miss the console feeling tho)
The problem is that modern "consoles" aren't consoles anymore - they're compact computers. Just as you wouldn't try to write a PC ethernet driver that ignored memory mapping, PS3 drivers shouldn't ignore memory mapping as well.
maaku
Posts: 4
Joined: Mon Aug 31, 2009 5:56 pm

Contiki OS?

Post by maaku »

mc, this is an interesting project. I would like to help you out, although the realities of my other commitments are such that I cannot make any promises.

Might I suggest though that instead of writing an operating system from scratch (as much fun as that might be) we should instead focus on porting an appropriately small, but functional embedded OS like Contiki? That way we can focus on providing PS3 API's rather than repetitive low-level tasks...
maaku
Posts: 4
Joined: Mon Aug 31, 2009 5:56 pm

Post by maaku »

Or MINIX, actually. That might be a better route to take than Contiki.
rapso
Posts: 140
Joined: Mon Mar 28, 2005 6:35 am

Post by rapso »

Why porting another OS if you can use linux?
I think the point was to have something lightweight from scratch dedicated for this one and only purpose: homebrew gamedev on ps3.
maaku
Posts: 4
Joined: Mon Aug 31, 2009 5:56 pm

Post by maaku »

Because when you look at what people really want for a homebrew development, Linux would not satisfy their needs. For homebrew you want an embedded operating system (if anything at all). Linux is a server OS. The two extremes couldn't be more different.

It would not be a very large undertaking. MINIX is already ported to the PowerPC, and the kernel itself is only 4000 lines of code.

EDIT: Another option (just to throw it out there) would be L4Ka::Pistachio.
rapso
Posts: 140
Joined: Mon Mar 28, 2005 6:35 am

Post by rapso »

maaku wrote:Because when you look at what people really want for a homebrew development, Linux would not satisfy their needs.
Thats simply not true. the linux kernel is very generic and exists in tons of versions,
all packages are optional, there are kinda rip-offs here at the forum that are already some basic game development os, the only issue ppl saw was the licensing via gpl, afaik.
For homebrew you want an embedded operating system (if anything at all). Linux is a server OS. The two extremes couldn't be more different.
there are plently of embeded linux versions, from realtime to pure virtual machine aimed like android.
It would not be a very large undertaking. MINIX is already ported to the PowerPC, and the kernel itself is only 4000 lines of code.
as far as i know, minix is just a micro kernel, so its probably not doing more than MC's current version of the gamedev os.
EDIT: Another option (just to throw it out there) would be L4Ka::Pistachio.
i think, as long as the license is compatible with the bsd alike license this new game dev os is made for, you can rip of any other os and "port" it to this one. but lets see what MC says:)
maaku
Posts: 4
Joined: Mon Aug 31, 2009 5:56 pm

Post by maaku »

rapso,

I'm sorry I don't have the time to engage in a full discussion on this, but let me offer at least a few points (from over a decade of experience in embedded software, linux kernel customization, and game development):

* Linux can be streamlined to an embedded profile, but at the cost of many man hours initially and continually in maintenance. Easy for a corporation to do, not so much for a loose-nit homebrew community.

* The existing kernels are all desktop/server derived variants of Linux. There is not, to my knowledge, any true real-time Linux kernel for the PS3/Cell. Real-time linux is always highly-coupled with the architecture that it is designed for, and developing such a kernel for PS3/Cell will take a non-trivial amount of time and resources to do properly. It's not just a matter of hitting some checkboxes and compiling.

* The ps3sdk has very small, but very specific needs that do not match up very closely with Linux (even with the patches Sony and those on this forum have made).

* Don't fall into the "Linux will be easier because it's Linux" fallacy. Linux is around ~3 million lines of code. MINIX is just 4000 lines of kernel code. L4 is of similar size, and already ported to 64-bit PowerPC. That's three orders of magnitude difference (yes, I know we are comparing micro- to macro-kernels, but for reasons of software engineering complexity it is relevant). What's more, these kernels already have reliable implementations of feature sets needed for real-time embedded programming and are simple enough to not require the extensive tweaking in the port that Linux RTOS options do.

Most people don't know, or underestimate even if they do the amount of work that must go into creating and maintaining a custom linux kernel for a real-time application. It is my professional opinion that the ps3sdk could be finished faster, and with better satisfaction by adopting an existing, smaller, non-Linux solution.

EDIT: But perhaps we should agree to disagree, if you're still not convinced. My free time is very limited, and any part of it I spend here would otherwise be spent on ps3sdk. And I think we all agree that flawed approach or not, any ps3sdk is better than no ps3sdk.


MC, are you still here? Unless I hear from you, I'm going to begin porting either MINIX or L4 to Cell, but I could use some of your expert knowledge... (or anyone else here that is familiar with kernel mode PPE).
Post Reply