The hunt for HV's FIFO/Push buffer...

Arwin · Post by **Arwin** » Sun Dec 23, 2007 11:37 pm

I have installed Fedora 7 and nothing else, but I'm not really good at development stuff under Linux yet. I've made some PSP projects in Cygwin, and that's about it.

But with good instructions or maybe just a pre-compiled testing program? I'd love to help some. I will have some time during the holidays at least. ;)

ps2devman · Post by **ps2devman** » Sun Dec 23, 2007 11:50 pm

Working tutorial for fedora 7 is coming (for noobs)... Stay tuned...
(I'm currently fighting in order to compile glaurung module with just kernel headers -something is missing in boxbuilder's tutorial-, but I won't give up...)
EDIT: Woot! Compiled! Now testing the remaining of boxbuilder's tutorial...
EDIT: 2D hardware acceleration successful! Tutorials PM'ed to oobles!

mitran · Post by **mitran** » Mon Dec 24, 2007 7:46 am

wot dose thies mean we have rsx? if so how on ubuntu?

ps2devman · Post by **ps2devman** » Mon Dec 24, 2007 6:55 pm

You have tutorials for ubuntu, debian, fedora 7 in the other forum thread :
http://forums.ps2dev.org/viewforum.php?f=27
(PS3 Linux developement)
See "PS3RSX Binary driver support" thread
If you find them too hard to follow, just ask politely their authors to rewrite a more noobs oriented version...

Glaurung · Post by **Glaurung** » Wed Dec 26, 2007 2:05 am

Hi all,

So it seems the hunt season has started again..

I have just noticed that the lv1_gpu_memory_allocate call ignores the highest bits (52-63) of the ddr_size argument in FW 2.01. So I have some little xmas holiday homework for people with FW >= 2.1. Please try changing the following in the ps3rsx module:

Code: Select all

        status = lv1_gpu_memory_allocate&#40;0, 0, 0, 0, 0,
...

to

Code: Select all

        status = lv1_gpu_memory_allocate&#40;1ULL << 60, 0, 0, 0, 0,

and report success or failure of loading the module.

Also, don't forget IronPeter's homework of checking what's in the FIFO after FB_SETUP in the standard ps3fb driver.

The closing of the gpu_memory_allocate hole was to be expected. However, from my tests, it seems the value 0 was clearly intended as a full access special value and not just a forgotten check. Anyway, we need to find a better way to create RSX objects.

The implications for now are that 3D is not available, but Xorg driver neither as it needs to create objects for accelerated solid fills and copy. I could make a unaccelerated Xorg driver with only Xv support, using the existing blitter object, but I'm not interested right now. I would rather focus on getting our 3D back ;-)

One thing that is still possible with FW > 2.1 is accelerated access to VRAM, for the mtd driver. I've been working on this recently and will post the corresponding driver update to the appropriate thread.

Merry Xmas everyone.

hermes · Post by **hermes** » Wed Dec 26, 2007 7:55 pm

Glaurung it allocate 0 bytes of DDR memory :O

Glaurung · Post by **Glaurung** » Wed Dec 26, 2007 9:11 pm

hermes: do you mean it does not return -17? If correct, this is great news as we basically have a fix for the 2.1 FW issue until it gets patched again. Can you paste the dmesg log?
Also can you check Xorg runs fine in this case?

hermes · Post by **hermes** » Wed Dec 26, 2007 11:10 pm

Glaurung wrote:hermes: do you mean it does not return -17? If correct, this is great news as we basically have a fix for the 2.1 FW issue until it gets patched again. Can you paste the dmesg log?
Also can you check Xorg runs fine in this case?

It return status=0 but the driver fail because it allocate 0 bytes... (the system hang) You obtain a valid handle and the ddr_lpar address.

I allocate 254MB after with other lv1 call and i can see it uses different handler (0x5a5a5a5b), but the same ddr_lpar

To be sure it alloc 0 bytes, i create two context and i can see 0 bytes for the first context and 0xfe00000 for the second context from the info datas

I can see it is possible to change the offset of the FIFO area to the last 2MB of the DDR (supposed) changing the GPU_IOIF value on my driver. It work but you cannot access to the RAMIN area using this trick and the RAMIN area is not mapped to other DDR address (i fill 254MB with 0 to test it). I can see in the crtl[0x10] register, the FIFO starts from 0xfe00000

Maybe is neccessary to activate a flag when you alloc a context to work in 3D

eldridge · Post by **eldridge** » Thu Dec 27, 2007 12:29 am

f/w: 2.10
kernel: linux-2.6.23-20071023
ps3fb: patched with ps3fb-fix-gpu-cmd-buff-size-2.10-linux-2.6.23-20071023

ps3rsx: lv1_gpu_memory_alloc(1ULL<<60,0,0,0,0,

dmesg
-------

ps3rsx: PS3 RSX access module, 1.0.0
ps3rsx: reserved XDR memory is @c000000000e00000, len 18874368
ps3rsx: 254MB of DDR video ram at 0x700190000000 mapped at d000080080780000 handle 5a5a5a5b
ps3rsx: context 0x55555554 dma=440000381000 driver=4000001a4000 reports=4800006d0000 reports_size=10000
ps3rsx: ctrl=d00008008010c000 drv=d0000800905c0000 reports=d0000800804e0000
ps3rsx: version 2.11 RSX rev9 0MB RAM channel 1 core 500MHz mem 650MHz
ps3rsx: remapped XDR apperture at c000000000e00000 size 18432kB to RSX

IronPeter · Post by **IronPeter** » Thu Dec 27, 2007 4:02 am

Glaurung, nice finding with lv1_context_gpu_attribute fifo_setup.

One note - you promised to make libps3rsx patch :).

Happy Xmas and New Year to all.

Rex_VF5 · Post by **Rex_VF5** » Thu Dec 27, 2007 4:27 am

Glaurung & IronPeter,

this is not regarding the topic however to me it seems the best place to ask (sorry if it's dumb): when in kernel mode wouldn't it be possible to access some part of memory that contains (unencrypted) firmware? Thre should be at least some portion that booted Linux etc.

Glaurung · Post by **Glaurung** » Thu Dec 27, 2007 5:17 am

IronPeter, I honored my promise, your christmas present is waiting for you in libps3rsx svn.

To FW 2.1 users, you can try this new libps3rsx version, with patched ps3rsx module as above (1ULL << 60 trick) and report if it works or fails. The dmesg log looks promising to me...

Rex_VF5, Linux does not have full access to the RAM, it is locked in a sandbox by the HV. Every 'physical' address used by Linux is not really physical but translated to the actual physical address under the control of the HV.

Note that the devices don't have full access either (through DMA), which is the reason the HV 'features' we exploit for enabling 3D under Linux are not really concerning security-wise (i.e. no Xbox360-like hack possible using shaders for example).

Rex_VF5 · Post by **Rex_VF5** » Thu Dec 27, 2007 6:00 am

Glaurung wrote:Rex_VF5, Linux does not have full access to the RAM, it is locked in a sandbox by the HV. Every 'physical' address used by Linux is not really physical but translated to the actual physical address under the control of the HV.

Note that the devices don't have full access either (through DMA), which is the reason the HV 'features' we exploit for enabling 3D under Linux are not really concerning security-wise (i.e. no Xbox360-like hack possible using shaders for example).

I didn't realize that. However it seems that it really works for the purpose it was designed ;-)

mitran · Post by **mitran** » Thu Dec 27, 2007 6:11 am

Glaurung wrote:IronPeter, I honored my promise, your christmas present is waiting for you in libps3rsx svn.

To FW 2.1 users, you can try this new libps3rsx version, with patched ps3rsx module as above (1ULL << 60 trick) and report if it works or fails. The dmesg log looks promising to me...

Rex_VF5, Linux does not have full access to the RAM, it is locked in a sandbox by the HV. Every 'physical' address used by Linux is not really physical but translated to the actual physical address under the control of the HV.

Note that the devices don't have full access either (through DMA), which is the reason the HV 'features' we exploit for enabling 3D under Linux are not really concerning security-wise (i.e. no Xbox360-like hack possible using shaders for example).

um i am a nobb but whear can i get the rxs patch? for 2.10 .or test it

IronPeter · Post by **IronPeter** » Thu Dec 27, 2007 6:29 am

Glaurung, very nice gift.

Rex_VF5, I did not try to run code at HV privilegies level or to hack HV. I see some possible holes.

I do not think that RSX DMAs are protected at the bus level. RSX can be theoretically used as a share resource by different instances of OtherOS. In that case the stream of DMA requests can not be splitted into independent parts.

I think that memory protection is provided by lv1_gpu_ioremap function. HV maps only memory from the current partition. Without RAMIN access it is secure design, because DMA objects are created by hypervisor. With RAMIN access we have read/write permission to all memory areas mapped to the RSX.

Probably there is some XDR HV memory mapped to the RSX inner address space. Probably, objects 0x1337808 and 0x13378086 target this memory. Note that xdr memory is mapped by large 1MiB chunks...

Panajev2001a · Post by **Panajev2001a** » Thu Dec 27, 2007 6:53 am

IronPeter wrote:I do not think that RSX DMAs are protected at the bus level.

According to the patent I quoted a few pages ago, they are (quoted text and then patent figure FIG. 1).

A memory access control apparatus executes memory access after the apparatus has determined whether access to memory is granted or not. A command receiver receives, from an external memory access requesting entity, a command with which to access data in memory, together with an address to be accessed and IOID to identify the memory access requesting entity. Based on the IOID, an access decision unit determines whether or not an access is one from a memory access requesting entity to which an access to a region designated as an access destination is to be permitted. The access decision unit determines whether access of the memory access requesting entity is permitted or not, for each page that serves as a basic access unit in memory.

http://appft1.uspto.gov/netacgi/nph-Par ... 2+AND+2007

I'd think that the memory range the IOID protects can be set at run-time and is not fixed at design/manufacturing though so if you have the right privileges you could attempt that. Somehow I think that the SPE set in isolation mode (the one kept aside for the HV/GameOS use) has to come into the picture though.

IronPeter · Post by **IronPeter** » Thu Dec 27, 2007 7:04 am

I've read this paper few pages ago.

Do not want to go into detailed discussion.

Panajev2001a · Post by **Panajev2001a** » Thu Dec 27, 2007 7:05 am

IronPeter wrote:I've read this paper few pages ago.

Do not want to go into detailed discussion.

I was not trying to insult your intelligence, just asking for your thoughts on it :).

Edit:

I think that memory protection is provided by lv1_gpu_ioremap function.

That function could be a nice candidate for what I was thinking about earlier: a function that gets the HV to change the mapping in that IOID unit.

Panajev2001a · Post by **Panajev2001a** » Thu Dec 27, 2007 6:08 pm

FW: 2.10
Kernel: 2.6.24-rc6-g8e36f255 (standard ps3fb.c and ps3fb.h)

dmesg's output (ps3rsx relevant part: when the system boots the kernel module is built and loaded using the ./load script minus the dmesg command which is executed by hand afterward):

Code: Select all

ps3rsx&#58; PS3 RSX access module, 1.0.0
ps3rsx&#58; reserved XDR memory is @c000000000d00000, len 9437184
ps3rsx&#58; 254MB of DDR video ram at 0x700190000000 mapped at d000080080780000 handle 5a5a5a5b
ps3rsx&#58; context 0x55555554 dma=440000381000 driver=4000001a4000 reports=4800006d0000 reports_size=10000
ps3rsx&#58; ctrl=d00008008010c000 drv=d0000800905c0000 reports=d0000800804e0000
ps3rsx&#58; version 2.11 RSX rev9 0MB RAM channel 1 core 500MHz mem 650MHz
ps3rsx&#58; remapped XDR apperture at c000000000d00000 size 9216kB to RSX

Why does not it map both RAM channels on RSX ?

So far so good, but... after this portion (the kernel modules is successfully loaded) the following error message is repeated a lot of times:

Code: Select all

ps3fb ioc0_01&#58; ps3fb_sync_image&#58; lv1_gpu_context_attribute FB_BLIT failed&#58; -24

In comparison dmesg used to look like this:

Code: Select all

&#91;root@IGGS-PS3 Panajev&#93;# dmesg | grep ps3rsx
ps3rsx&#58; PS3 RSX access module, 1.0.0
ps3rsx&#58; reserved XDR memory is @c000000000d00000, len 9437184
ps3rsx&#58; 254MB of DDR video ram at 0x700190000000 mapped at d000080080780000 handle 5a5a5a5b
ps3rsx&#58; context 0x55555554 dma=440000381000 driver=4000001a4000 reports=4800006d0000 reports_size=10000
ps3rsx&#58; ctrl=d00008008010e000 drv=d0000800905c0000 reports=d0000800804e0000
ps3rsx&#58; version 2.11 RSX rev17 0MB RAM channel 1 core 500MHz mem 650MHz
ps3rsx&#58; remapped XDR apperture at c000000000d00000 size 9216kB to RSX
ps3rsx&#58; context 0x55555557 dma=440000382000 driver=4000001a8000 reports=4800006e0000 reports_size=10000
ps3rsx&#58; ctrl=d00008008011c000 drv=d000080090600000 reports=d0000800805e0000
ps3rsx&#58; version 2.11 RSX rev17 0MB RAM channel 2 core 500MHz mem 650MHz
ps3rsx&#58; remapped XDR apperture at c000000000d00000 size 9216kB to RSX

Glaurung · Post by **Glaurung** » Fri Dec 28, 2007 12:36 am

Interesting..

The 'channel' number does not correspond to 'RAM channel', it is the hardware context number (see http://nouveau.freedesktop.org/wiki/ContextSwitching). The ps3rsx module creates a context for itself on startup, and then creates another context when a user mode application opens /dev/fb1. So by simply loading ps3rsx you should only see one 'channel' being reported and it should be 1 because ps3fb allocates its own channel and thus already uses channel 0. When an application such as libps3rsx or Xorg uses the driver, it should use channel 2 and you will see the second set of lines in dmesg.

The -24 error means we broke the ps3fb FB_BLIT call (it is reported each time the ps3fb driver copies its virtual framebuffer in video ram => often). Usually this happens because the GPU is in incorrect state due to previous tweaking.

If you have time, can you trying commenting out some functions in the ps3rsx module to see which one causes the ps3fb blit to fail? I would start by commenting out the context creation, line 732-735. Then if it works, reenable it, and comment inside this function, starting line 392-436 and progressively enabling more stuff. This would give some insight on what the lv1_gpu_memory_allocate(1UL << 60, ...) really did. Note, on FW 2.01, it does exactly the same as calling lv1_gpu_memory_allocate(0, ..), which is why I thought it was worth trying.

Panajev2001a · Post by **Panajev2001a** » Fri Dec 28, 2007 1:04 am

I'll try to do something tonight, I am not going to be using the PS3 much if at all in the next few days, but still it could be an useful exercise :). I think that others did not report this error because they are running a kernel with an older ps3fb and so nothing ended up being broken by ps3rsx.

eldridge · Post by **eldridge** » Fri Dec 28, 2007 2:29 am

FB_BLIT -24 error on my configuration as well was testing on ssh con and console had got blanked so no BLIT.

As soon as I started X or did on tty got the FB_BLIT error.

I was just wondering if this call now interfered with the new cmd sized buffer in ps3fb that is now setup in the ps3fb-fix-gpu-cmd-buff-size under 2.10.

If it altered this buffer in some way it might have disrupted the blit submission which goes into this buffer? Will look more and attempt to get up to speed with the HV calls.

Might try a couple of other safe shift values for the 1ULL<<60

BTW glaurung thanks for the libps3rsx update.

monolith42 · Post by **monolith42** » Sat Dec 29, 2007 12:41 am

lv1_gpu_context_attribute is the point of failure for ps3rsx.

Going to try to move the FIFO to the start of the apperture (like the kernel patch) to see if that helps.

monolith42 · Post by **monolith42** » Sat Dec 29, 2007 1:06 am

If i change the arguments of lv1_gpu_context_attribute so wptr and rptr are the same then nothing breaks (nothing gets done either).

Looks like it's the fifo_setup_program that breaks stuff.

--
If i only execute everything up to the 2D surfacecontext part it still works without breaking.

--
If i remove the 2D surfacecontext part from the module is loads succesfully without breaking FB_BLIT.

monolith42 · Post by **monolith42** » Sat Dec 29, 2007 1:28 am

And now for the bad news.

After fixing so the module loads succesfully i tried the examples in libps3rsx.
They comple and run but there is no 3d visible on screen (the screen is however 'corrupted' until the program is terminated, it is restored properly after termination).

IronPeter · Post by **IronPeter** » Sat Dec 29, 2007 6:17 am

monolith42, did you run init script src/init/mknods.sh ?

Success of that initialization is 90% in memory access rights and limits for DDR context memory.

monolith42 · Post by **monolith42** » Sat Dec 29, 2007 6:58 am

Yes i've run that script succesfully.

monolith@ps3-linux ~/source/libps3rsx/src/init $ sudo ./mknods.sh
Password:
make: `init' is up to date.
3d engine setup completed

---
Please note that my ps3rsx module is missing the 2d surfacecontext subchannel object (it breaks FB_BLIT if left in its current state).

IronPeter · Post by **IronPeter** » Sat Dec 29, 2007 7:21 am

Pretty nice, can you make RAMIN dump? Sources are in this thread.

monolith42 · Post by **monolith42** » Sat Dec 29, 2007 7:25 am

Here's the pushbuffer dump right after FB_SETUP

ps3fb: offs: 00000000, val: 00042000
ps3fb: offs: 00000004, val: 31337303
ps3fb: offs: 00000008, val: 00042180
ps3fb: offs: 0000000c, val: 66604200
ps3fb: offs: 00000010, val: 00082184
ps3fb: offs: 00000014, val: feed0001
ps3fb: offs: 00000018, val: feed0000
ps3fb: offs: 0000001c, val: 00044000
ps3fb: offs: 00000020, val: 3137c0de
ps3fb: offs: 00000024, val: 00044180
ps3fb: offs: 00000028, val: 66604200
ps3fb: offs: 0000002c, val: 00084184
ps3fb: offs: 00000030, val: feed0000
ps3fb: offs: 00000034, val: feed0001
ps3fb: offs: 00000038, val: 00046000
ps3fb: offs: 0000003c, val: 313371c3
ps3fb: offs: 00000040, val: 00046180
ps3fb: offs: 00000044, val: 66604200
ps3fb: offs: 00000048, val: 00046184
ps3fb: offs: 0000004c, val: feed0000
ps3fb: offs: 00000050, val: 00046188
ps3fb: offs: 00000054, val: feed0000
ps3fb: offs: 00000058, val: 0004a000
ps3fb: offs: 0000005c, val: 31337808
ps3fb: offs: 00000060, val: 0020a180
ps3fb: offs: 00000064, val: 66604200
ps3fb: offs: 00000080, val: 313371c3
ps3fb: offs: 00000084, val: 0008a2fc
ps3fb: offs: 00000088, val: 00000003
ps3fb: offs: 0000008c, val: 00000004
ps3fb: offs: 00000090, val: 00048000
ps3fb: offs: 00000094, val: 31337a73
ps3fb: offs: 00000098, val: 00048180
ps3fb: offs: 0000009c, val: 66604200
ps3fb: offs: 000000a0, val: 00048184
ps3fb: offs: 000000a4, val: feed0000
ps3fb: offs: 000000a8, val: 0004c000
ps3fb: offs: 000000ac, val: 3137af00
ps3fb: offs: 000000b0, val: 0004c180
ps3fb: offs: 000000b4, val: 66604200

monolith42 · Post by **monolith42** » Sat Dec 29, 2007 7:31 am

Could you possibly pm the RAMIN dump sources?

Searching this thread is a pain because of its size.

forums.ps2dev.org

The hunt for HV's FIFO/Push buffer...

gpu_memory_allocate ignored bits

libps3rsx using ps3rsx module

Re: libps3rsx using ps3rsx module

Re: libps3rsx using ps3rsx module

Point of failure