How to get more RAM for the PS3 under Linux. My hypothesis.

Nismobeach · Post by **Nismobeach** » Thu Aug 16, 2007 2:44 pm

Let me preface this thread by saying that I'm fairly ignorant when it comes to programming, but I am fairly knowledgeable about basic computer hardware, operating systems, and basic o/s functions. In any case I apologize in advance for any shortcomings in what I'm about to propose and welcome any and all criticism and feedback. :)

We are all aware that the 256MB of system RAM is one of the biggest bottle necks that limits the usability of Linux on the PS3. Furthermore most are aware of the limitations of utilizing the GDDR3 video memory for use via Linux as well. Thus the quagmire of a question that remains is how does one go about upgrading the usable system RAM on a system that isn't upgradeable?

Awhile back I read an article about how Ken Kutaragi justified the inclusion of Gigabit Ethernet on the PS3 because of claims that clusters of CELL processors would be able to work together to "sharpen" content. A real world application of this can be clearly seen in the PS3 real time ray tracing video which can be found here.

It appears that the only true way to enhance the PS3's capabilities is via networking it to other consoles, or other machines...

Thus, my hypothesis is that the most viable way to expand the RAM (amongst other things) on the PS3 is by utilizing the built in Gigabit Ethernet to network it to a "server" (via gigabit components) with an i-RAM solid state hard drive and using said hard drive for swap/virtual memory functions under Linux on the PS3.

Possible advantages:
Since the i-RAM uses cheap DDR SDRAM it would be cheap to purchase and build a 4GB unit.

i-RAM detects and is accessible like a normal hard drive under a PC bios and computer operating systems.

i-RAM performs extremely fast when compared to conventional hard drives and has great random read/write capabilities.

Could potentially create interest into exploring the possibilities of networking to other "server side" devices.

Possible disadvantages:
i-RAM performance in a PS3 application could possibly be limited by PCI bus/bandwidth and Gigabit speed.

Unsure of the system requirements on the server side to run such a setup.

It might not even be possible...

The bottom line:
In my view I believe that after reading this article titled RAM Disks over Gigabit Ethernet the i-RAM setup mentioned above would work for swap/virtual memory and yield decent performance. Furthermore since the i-RAM would be superior in its read/write capabilities it should exceed the performance of conventional hard drives used in the latter link that I posted as well as the PS3's own hard drive.

All things considered I estimate that the proposed i-RAM/Gigabit Ethernet/cache server setup would be able to yield net read/write capabilities in the 125 to 140 MB/s range. Not great, but at least it's a start.

Discuss. :)

jimparis · Post by **jimparis** » Thu Aug 16, 2007 5:31 pm

Nismobeach wrote: i-RAM performs extremely fast when compared to conventional hard drives and has great random read/write capabilities.

It's not that much faster than a hard drive -- maybe 2-3x. Also, there's no reason at all to use the i-RAM card -- just get a server, stick 4GB of RAM directly in, and use that.

But anyway, gigabit is painfully slow compared to real RAM: Cell to main RAM bandwidth is somewhere around 25 GB/sec, and LAN-based memory would be limited to around 100MB/sec. Sure, it might be 2-3x faster than HD-based swap, but if you're hitting swap hard with your workload, your performance is already shot -- I'm not sure 2-3x is gonna help much in that case.

theBishop · Post by **theBishop** » Wed Sep 26, 2007 12:40 am

wouldn't USB flash memory be faster than this?

jimparis · Post by **jimparis** » Thu Sep 27, 2007 1:02 am

No, not by a long shot.
USB flash drives don't get more than ~35MB/sec on reads.

episch · Post by **episch** » Tue Oct 09, 2007 2:42 pm

Is there anything in the hardware stopping someone from say, desoldering the RAM (hot air?) and then using the toaster oven reflow method to put larger capacity chips on there?

jimparis · Post by **jimparis** » Tue Oct 09, 2007 4:46 pm

That won't happen.
1) You're gonna have a hard time finding the right Rambus XDR ram chips.
2) There's no reason for them to have wired up extra address lines
3) The hypervisor wouldn't know to give the extra RAM to lv2 and we can't change that.

episch · Post by **episch** » Tue Oct 09, 2007 6:48 pm

jimparis wrote:That won't happen.
1) You're gonna have a hard time finding the right Rambus XDR ram chips.

Ignoring your last two points, I wasn't even able to source denser XDR chips then the 512Mb that are on the board... so at least for now it's not even feasible from that aspect.

Edit: Thank you for the input.

linux_my_toaster · Post by **linux_my_toaster** » Tue Oct 30, 2007 4:32 pm

What about using SATA? Should be around 3 times as fast as the gigabit ethernet connection...

The hypervisor shows linux a fake drive, right? But that fake drive is mapped to a real location on the disk... so if you could emulate a hard disk on the other end of the ps3's SATA cable, but map the last gig or so of the linux partition to a ramdisk... linux on the PS3 should still be able to access that portion of the disk indirectly through the hypervisor, mount it as a ramdisk and use it as additional high speed swap.

so it would be like....

PS3linux -> hypervisor -> emulated SATA disk on server -> ramdisk -> physical ram on server

where that last gig or so is indistinguishable from the rest of the drive, just ridiculously fast. Do I make any sense?

Of course, it's still slow compared to ram, but it would be a fast swap, and 3x as fast as ethernet, if I'm not mistaken. thoughts?

Edit:

There used to be pci cards that used ram as a disk and kept the whole thing powered. Donno if those exist in SATA form, but that would be a hell of a lot simpler. doh

Edit 2:

haha... you could use raid between the SATA ram and the ethernet ram, with every third bit on the ethernet (: ~400MBps?

rapso · Post by **rapso** » Tue Oct 30, 2007 4:57 pm

Back in the windows 3.11 days (don't shoot me ;) ), there were memory manager that compressed memory on the fly. That memory could be decompressed on the fly when a pagefault occurs. (compressing could be done lateron with an SPU with no slowdown for linux).
Games usually use comressed files to speedup loading and saving as well, so we can expect that we could not only virtually make the usable memory bigger, but also speedup the swapfile-access-times.

I would voluntee me to write that little fast compressing lzw or huffman code for ppu first but with little local footprint to be then used on spu.
But i'm not familar with linux memory management sources (especially not for PS3), so someone else would need to make that.

Tell me if it's a stupid idea ;)

linux_my_toaster · Post by **linux_my_toaster** » Tue Oct 30, 2007 5:28 pm

hm... if you used an SPE to do the compression work transparently on the fly and presented linux with more ram artificially... how much would the speed sacrifice be?

I'm not familiar with the compression ratio lzw can yield on binary data... around 2-3 possibly? So perhaps you could double the apparent physical ram? Interesting indeed

ps2devman · Post by **ps2devman** » Tue Oct 30, 2007 5:29 pm

Be sure to follow progress made by IronPeter and Glaurung here :
http://forums.xbox-scene.com/index.php?showtopic=624724

With that kind of knowledge you can fill in textures in the 252Mb (gpu side ram) and have RSX display them at will and fast.

That way you have the 256Mb (cpu side ram) for all non graphic stuff.

Also, mass and fast data processing can be done though GPU shaders by just considering textures as arrays of data.

unsolo · Post by **unsolo** » Tue Oct 30, 2007 5:30 pm

Crazy stuff:
Render X onto a jpg Screens using existing variable length coding patterns
Decompress the screens onto the framebuffer in a "ordered" manner.

:)

rapso · Post by **rapso** » Tue Oct 30, 2007 6:33 pm

linux_my_toaster wrote:hm... if you used an SPE to do the compression work transparently on the fly and presented linux with more ram artificially... how much would the speed sacrifice be?

I'm not familiar with the compression ratio lzw can yield on binary data... around 2-3 possibly? So perhaps you could double the apparent physical ram? Interesting indeed

This heavely depends on the data.

Regarding to Statistics, a lot of memory is filled with just 0 or 0xff. (IBM has even a patent for some memory-controller that use this information for compression!). for example

Code: Select all

uint64_t a=1;

will have like 7 0x0 and one 0x1 bytes.
In lucky moments you might have a very good compression, in bad moments no compression.

LZW and Huffman can be very fast, it's mostly a performance vs memory-usuage vs compression-ratio. I did not benchmark on ppu, but on a 2GHz AMD64 you can easily get 40MB/s with conservative memory usuage.

The hard thing is probably getting the memory management smart to have speed improvements. e.g. it would be of no use to compress on page from 4kb to 3kb if you need to access on harddisk-page that is 4kb in size. so you need to e.g. just save it compressed if there are two ram-pages compressed into one 4kb harddisk-page etc.

unsolo · Post by **unsolo** » Wed Oct 31, 2007 10:02 pm

if you pack using a binary compressed version of a spu_shuffle pattern
values are 0-31 and the 0xFF 0x80 and 0xc0 unique values and you have 16 slots on where to insert it.

With a certain data ordered to your best compressed liking. you should be able to compress and decompress lossless large amounts of any data with little effort.

challange is "best reordering" something not to far from a dynamic zigzag.
lookup time for the pattern.

using: data -> Huffman + shuffle_pattern -> data

may be im way off but thats my thoughts on the subject

ralferoo · Post by **ralferoo** » Thu Nov 01, 2007 10:00 am

rapso wrote:I would voluntee me to write that little fast compressing lzw or huffman code for ppu first but with little local footprint to be then used on spu.

Someone already released an SPU optimised zlib replacement earlier this year: http://www.power.org/forums/viewtopic.php?t=1014

linux_my_toaster · Post by **linux_my_toaster** » Fri Nov 02, 2007 6:58 pm

always try google last, that's what I always say...
/sarcasm

Firstly, CRAMES is meant for embeded systems but should do what we're talking about. It does seem to assume that a large portion of memory would be occupied by latent applications, but this is only one issue it addresses.

also:

http://linuxcompressed.sourceforge.net/

At the least, significant bits of the code from either of these projects could probably be used along with pre-existing SPU decompression code...

boxbuilder · Post by **boxbuilder** » Sat Nov 17, 2007 4:05 pm

I have tried swapping over network, First I made a ramdisk on my winblows box, then I shared it using smb, and mounted the smb share to /ramdisk then in /ramdisk I made a swap file, and prepped it for swap, but when I used the swapon command, it refused. So i tried nfs, same result. I have since found that there are kernel patches to support swap over network
http://kerneltrap.org/mailarchive/linux ... /30/369602
but I can't figure which revision is for which kernel. At any rate, the patches in http://programming.kicks-ass.net/kernel ... .6.23-mm1/
http://programming.kicks-ass.net/kernel ... k/current/
don't work with either geoff's kernel
git://git.kernel.org/pub/scm/linux/kernel/git/geoff/ps3-linux.git
that kernel http://www.kernel.org/pub/linux/kernel/ ... 23.tar.bz2
or the kernel from the latest cell disk CELL-Linux-CL_20071023-ADDON.iso
another way would be to share a block device as described on http://nbd.sourceforge.net/ but I don't have a linux box sharing the ramdrive.

Hardware solutions: if $ony just put one extra sata plug on their motherboard it would be Christmas. As it is, i think the best chance of getting added ram/swap would be to buy a sata ramdrive, and a sata hub, or a sata2usb adaptor. The hub may not work, and the usb will never beat half a gigabit (but seek times will be excellent). I don't know much about the ps3 sata controller so using a sata hub might not work or it may need to be hardware raid.

audi100quattro · Post by **audi100quattro** » Mon Dec 10, 2007 2:22 pm

As long as compression+decompression of a page is faster than a seek on the swap HD block device, compression is a performance gain. Whether you keep the compressed data in RAM or write to swap is secondary. The problem seems to be in finding the right things to compress with the right algorithm, storing the metadata for the compressed pages, and dealing with pages where the sought data might be distributed. You also read less from swap every time you compress something which brings its own gain.

The linuxcompressed.sf.net site linux_my_toaster linked to seems to have some prototype code for 2.6.16 kernels with the problems laid out... http://linux-mm.org/CompressedCaching
The zlib port using spe's would help a lot here. Memory pages for a program like firefox have to be compressible quite a bit...

Failing that, I'm doing a raid0 with fast USB sticks before swapping to network. The latency will be a lot lower with the USB sticks.

Just a couple thoughts for now since I'm new here :)

navimarin · Post by **navimarin** » Wed Dec 12, 2007 2:30 am

Mounting a USB stick or a MemoryCard (reader is included in PS3 isn't it?) as swap should be fine for running linux shouldn't it? All that matters here is latency. This way the swap wouldn't be that slow right? It should at least be faster as swapping to hd.

jimparis · Post by **jimparis** » Wed Dec 12, 2007 2:45 am

Speed matters a lot.

audi100quattro · Post by **audi100quattro** » Sun Dec 16, 2007 11:50 am

Has anybody used ata over ethernet for swap? the ps3 Kernel should treat it as a block device giving it the priority it needs right?

jimparis · Post by **jimparis** » Sun Dec 16, 2007 3:16 pm

Any sort of swap over network is tricky, because the network code may need to do memory allocations which can lead to deadlocks.

Nismobeach · Post by **Nismobeach** » Tue Dec 18, 2007 2:20 pm

I'm glad to see that my thread has sparked some debate. With regards to swap memory, I understand that using a flash card has it's limitations as the number of writes would eventually kill the card, and that swap over ethernet might not be quick enough.

It got me thinking though. How about using a compact flash style Seagate microdrive instead.

http://www.amazon.com/Seagate-CompactFl ... 35-4729710

Technical Details

* Hardware Platform: PC
* device_type: Hard drive
* storage_controller: None
* storage_controller_interface: CF+
* Hard Disk Description: Removable hard drive
* Hard Disk Size: 8 GB
* Hard Disk Rotational Speed: 3600
* Buffer Size: 2 MB
* Width: 0.02 inches
* Depth: 1.4 inches
* Height: 1.0 inches
* Weight: 0.2 pounds
* Warranty: 1 year warranty
* computer_peripheral_connector: 50 pin IDC

It seems like this would be a decent solution to people who have a 60/80GB users as long as it is a quick(er) alternative to what has been suggested above.

jimparis · Post by **jimparis** » Tue Dec 18, 2007 4:05 pm

Reviews suggest that the ST68022C-RK microdrive has a read performance of 8MB/sec and write of 2.5MB/sec -- easily an order of magnitude worse than the SATA hard drive. So that is also not a good idea.

Jim · Post by **Jim** » Tue Dec 18, 2007 4:27 pm

Flash isn't so bad an idea. It's so cheap that even if you needed to replace it at the rate of 1Gb a month it'd still be cheap. And by then the next generation of flash will be out with twice the number of write cycles.
Sandisk IV is up to 40MB/s.

Jim

jimparis · Post by **jimparis** » Wed Dec 19, 2007 3:36 am

SATA hard drives are cheaper than flash, have no write cycle issues, and can easily reach 40MB/sec. So why bother?

Nismobeach · Post by **Nismobeach** » Wed Dec 19, 2007 7:56 am

Jim wrote:Flash isn't so bad an idea. It's so cheap that even if you needed to replace it at the rate of 1Gb a month it'd still be cheap. And by then the next generation of flash will be out with twice the number of write cycles.
Sandisk IV is up to 40MB/s.

Jim

That is a good point. I think I remember seeing a 1GB SD card at Walmart today for only $12.

jimparis · Post by **jimparis** » Wed Dec 19, 2007 8:56 am

Nismobeach wrote:That is a good point. I think I remember seeing a 1GB SD card at Walmart today for only $12.

Huh? Does nobody listen? SD would suck for swap. Max speed for SD 1.1 = 166x = 20MB/sec, and benchmarks show most SD cards falling below 10MB/sec.

voney · Post by **voney** » Wed Jan 02, 2008 8:29 pm

Hi newbie here.

Perhaps something like this (http://www.bigbruin.com/reviews05/thumbraid_1) would be in order...

mattruby · Post by **mattruby** » Mon Jan 28, 2008 2:47 am

voney wrote:Hi newbie here.

Perhaps something like this (http://www.bigbruin.com/reviews05/thumbraid_1) would be in order...

What about making the PS3 Linux boot from a network drive over the 1Gbps ethernet (which wouldn't be a lot slower than the SATA), and then replacing the SATA drive with an i-RAM with 4GB, mapped as a swap. That would give the system a lot of RAM, even though it wouldn't be directly addressable (the swap manager would page it in and out of system RAM).

But the Linux memory manager could be changed to use the swap manager access to the i-RAM, letting malloc() use i-RAM transparently. Slower than onboard RAM, (max 1.5Gbps rather than 25Gbps), but usable as IO data pools accessed by PPC apps directing chunks to the fast SPUs, which don't access main memory as often as their algorithms access their local RAM.

FWIW, the i-RAM is "PCI" only for power supply, which could be supplied by an external enclosure containing a PCI riser.

Also, does anyone know whether the PS3 has 4 independent USB buses, each with its own 480Mbps bandwidth, or whether it's got a single (or just two, etc) USB bus with all ports sharing the 480Mbps? If it's really got 4 buses, that's another 1.92Gbps that a Flash RAID could use. If the memory/swap OS code could be hacked to address the USB Flash RAID and the SATA i-RAM in parallel, but present it as just malloc()'able memory, that could be 3.42Gbps "RAM", with something like 20GB attached.

Anyone have any real showstoppers on this strategy? Anyone up to hacking the memory/swap managers?

forums.ps2dev.org

How to get more RAM for the PS3 under Linux. My hypothesis.

How to get more RAM for the PS3 under Linux. My hypothesis.

Re: How to get more RAM for the PS3 under Linux. My hypothe

swap over network