forums.ps2dev.org

[UsefulIdiot]

I have been having some problems with random crashes lately. When I say random, I mean during random builds, my game will just crash. What is strange, is removing or adding random printf's will make it go away for a few builds. If a build does not crash once, it will run fine, if it crashes once, itll crash always.

Heres what psplink tells me when it crashes:

[dr_watson]

Most of the time when I got crashes like that was due to NULL pointers.

[groepaz]

[hubevolution]

ehehe well that's a very NULL pointer :)

[Brunni]

If it just happens sometimes, then it's most likely a bug in your program, like a buffer overflow.
Sometimes it can also be a bug in GCC.
_________________
Sorry for my bad english
Oldschool library for PSP - PC version released

[Saotome]

...but without any code we will never know :P
_________________
infj

[turkeyman]

I've actually been experiencing the same thing..
I build and get a random crash in a bit of code thats always run just fine..
Then i start narrowing down the crash, isolate the function its in, then put in some logs to see why its crashing, and it stops crashing..

Its happened to me in 2 places that wouldnt normally crash... and it went away when i put logs in the code between specific lines..

I'm a little suspicious of something in a recent revision of the toolchain.. Its almost like GCC is scheduling the code badly..

I had another bug where i had an assert that was firing off because something was an invalid value, so i put logs around it to see what the value was, and it was definately the correct value, and the code around it that depended on it being correct was still working perfectly.. It appeared like a scheduling thing in the assert test.. like the value hadnt been set before the assert test or something....

:/

[starman2049]

I've been fighting this every day for two months. Every time I make a code change I have to add/subtract unused lines of sprintf's (outside my main loop) so that it does not crash and/or does not have graphic corruption.

I'm pretty sertain it has to do with alignment, because 90% of the time it just gives graphic corrution and only 10% does it actually crash. I'm not using PSPLINK so I don't get and crash dump info, but I'm thinking more and more about converting to PSPLINK, but was hoding off until after E3.

I got the sdk and toolchain on September 5, 2005 and have been too nervous about screwing up my dev environment to get a more recent version so at least for me I cannot blame sdk or toolchain.

I've spent at least a week trying to fix this, but was never able to track it down. If anybody finds a solution that works for me I'll buy you a beer, heck I'll buy you a whole keg and that's no joke.

[turkeyman]

I dont know if that sounds like that same thing :/

I have had numerous problems with alignment too, but this is very different..
Its totally unsolicited, and if you wiggle the instruction order around it just go's away.. also logging the values shows the wrong thing, when i know the right value is in the variable..

[cheriff]

Random bugs that are there one minute and gone the next are always (in my experience) one of two things:
* Timing issues and/or race conditions. However since were arent running smp (unless your using the ME in which case you're on your own! :) and not in a preemptive environment, we can pretty much skip this case, leaving us with:
* Cache issues. Could very well be the case. When reading back from the cpu you get the correct value - since the cpu is able to read the most current one from the cache. Kicking the buffer to the gu happens behind the cache and perhaps there's a bad value in ram causing a bus error when dereferenced? Adding random loggings, moving code around also helps as it changes timing/cache usage patterns and gives the data a chance to eventually be flushed back to main ram.

I don't know if this is the case, but there may be just one instance you forgot to writeback the cache. Its what the symptoms seem to indicate and what I'd be looking for..

Good luck!
- cheriff
_________________
Damn, I need a decent signature!

[turkeyman]

it sounds like that could be the case for starman2049, but i dont think my or UsefulIdiot's cases are likely the dcache..

Heres one of multiple examples (also remember they both used to work perfectly..

const char *MFStringCache_Add(MFStringCache *pCache, const char *pNewString)
{
MFCALLSTACK;

MFDebug_Assert(pCache, "NULL String cache!");
MFDebug_Assert(pNewString, "Cannot add NULL string");

// find the string
char *pCurr = pCache->pMem;
int newLength = MFString_Length(pNewString)+1;
printf("%p %p, %s\n", pCurr, pCache, pNewString);

while (pCurr[0] && pCurr < &pCache->pMem[pCache->size])
.....
}

heres the output when i run with the printf moved up one line (it dies on the 4th iteration):
host0:/> 0x08DBD18C 0x08DBD180, Material
0x08DBD18C 0x08DBD180, Arial
0x08DBD18C 0x08DBD180, diffusemap
0x616D6573 0x08DBD180, Arial <- notice the value of pCurr here
Hardware Exception!

you can see where i have placed the printf, this code now works..
with that printf where it is, all 3 values show correctly and the code works perfectly..
if i move that printf to be on the line before the strlen (still after pCurr=...), pCurr changes and is no longer valid (but pCache is still the same pointer), and remains that way until the while() line where it crashes because it cant dereference..

I've toyed with loads of dcache flushes and stuff, doesnt seem to help..

:/

this is one of multiple similar examples.. i can provide some others if you like.

[turkeyman]

oh, also worth noting, if i take the printf out altogether, it crashes exactly as in my example there.. but if i compile using -O0, it works..

huff... (another reason i started considering the possibility of it being gcc's scheduling).

[turkeyman]

infact, -O0 and -O1 both work.. -O2/-Os/-O3 all break it..
what optimisation techniques are introduced between -O1 and -O2?

[Jim]

I suspect it's more likely that you have overwritten the stack. Optimisation often pushes variables from the stack into registers, and adding and removing lines of code often moves where things are on the stack too. Perhaps you have something like

char x[10];
sprintf(x, "DeadDeadDead");

or some other stack smashing code.

Jim
_________________
http://www.dbfinteractive.com

[weak]

i've experienced graphical errors with -O2 too (while -Os works perfect).

chp told me that he made some fixes regarding this problem, and a toolchain update indeed corrected _most_ of the bugs.

but seems like there are still some problems with -O2

[turkeyman]

Jim:

I dont think thats likely, i have made fairly thorough searched for buffer overflows..
Also, from my example, pCurr, and pCache are the only variables on the stack.. pCache is correct, pCurr is broken, and strlen() certainly doesnt write to the stack.. so i cant see any place where it could be overwritten...

:/


weak:
He said '_most_'? .. where was this announcement made?
Are there more known bugs, and do you know if anyones sorting them out?
I know nothing about GCC internally, so i'm not much use there.. :/

[weak]

[Brunni]

Sorry to bump this topic, but are still people getting problems like this? I downloaded the latest version of the toolchain but I'm still having those problems :(
I've looked at my code carefully and it seems to be okay (althrough so complicated that I can't be 100% sure), it also runs under Windows in debug mode without problem, so chances that there's a stack overflow or anything in it are very small...
However, in some builds I'll get random crashes (game freezed, sound buffer is looped), in some others temporary graphic corruption (more rare). If I change anything in the code, it won't happen again... so it's almost impossible to debug...

I really want to find where the problem could be, but I haven't got any tool for this. Can some people tell me what I could do for debugging? I'm thinking to dump changes in RAM from one function to another to see where an eventual overflow could occur...

Thanks in advance
_________________
Sorry for my bad english
Oldschool library for PSP - PC version released

[jimparis]

[Brunni]

No I cant :( But Visual Studio's debug mode checks for that quite good usually, I think ^^
I'm still getting those errors, it seems only in -O2, -Os or -O3 mode, and very strange things happen, for example my menu flickers at first (it should not be drawn), then should fade in => freezes at 50%. And when these bugs happen, it always freezes (I have installed a debug exception handler and it doesn't appears in these cases). Usually when it's my fault, the exception handler is fired.
That's why I'm wondering if it could more be a problem with GCC. But am I alone? Is this an official MIPS-GCC or is it tweaked (and so maybe a bug kicked in)?
Thanks in advance
_________________
Sorry for my bad english
Oldschool library for PSP - PC version released

[phytoporg]

Sorry to revive an older thread, but I'm running into the same problems. Seems like everything is working properly, but when I remove arbitrary pieces of debug output, the program just crashes with nothing from the exception handler.

Toying with the gcc flags doesn't seem to make any difference. Anyone have any other suggestions? This is crazy frustrating. D:

[Jim]

The answers are the same, either you've
a) Overwritten the end of a memory allocation (corrupted the heap)
b) Overwritten the end of a local array (corrupted the stack)
c) You're not handling the PSP's cache correctly when addressing the video hardware.
d) Something else.

Changing the optimisation flags often moves things around on the stack and re-orders the code to make things just slightly different, same as removing of adding the odd debug statement.

No code. Not much anyone can do to help.

Jim
_________________
http://www.dbfinteractive.com

[phytoporg]

I'd provide some source but I'm not entirely certain where the problem is. I'd like to try PSPLink's gdb capabilities to narrow things down but I'm using 2.6 firmware. I'm actually not even 100% sure where this thing's crashing since I can't make use of an exception handler.

I'm attempting to write a dynamic recompiling/threaded interpreting SNES emulator (I'm not really clear on the distinction). My current problem is that when I comment out the code to dump blocks from the instruction cache-- the emulator's, not the PSP's, obviously-- to the memstick, the program seems to crash upon attempting to execute the generated code. If I leave the logging in there, I don't get any problems unless I completely comment out all of the other printf stuff that's hanging around there.

I'll post what I think could be relevant code:

from compileBlock():

[ector]

phytoporg, are you invalidating the instruction cache before trying to execute newly generated code? Doing this is essential.
_________________
http://www.dtek.chalmers.se/~tronic/PSPTexTool.zip Free texture converter for PSP with source. More to come.

[phytoporg]

Yeah, I was talking to another developer yesterday who suggested the same thing. It seems to have fixed the issue. I'm now also forcing a dcache writeback after the block is "emitted." Everything's in working order, now!

[Aion]

I've been trying to release a new version of PspKanji for a month, and I'm having a similar problem which is stomping me at the moment :/.

The code works just fine under Windows/DirectX.

I've read about the possibility of stack corruption, but I am unable to pin point anything that could cause it.

On the Psp, it will crash at different moment, depending on code changes and wether I'm running the .pbp or the elf with PspLink or under Gdbdebuguer/Eclipse. Compiler optimisation options also changes moment crash occur.

I've attached the source code of the project if anyone would be interested in helping me out on this. http://www.pqverdun.org/SvnPspKanji.rar

Steps :
1-Start the app
2-Press any key, ressources will load
3-Press Triangle and select Kanjilist1.txt (confirm with X)
4-Press X to start Quiz
5-Press O to quit Quiz
6-Repeat 4,5 until app crash.
(Note sometime crash on loading KanjiList, something when trying to load quiz for 1st time, some other time after 3 quizes...)

Possible result :

[Aion]

When debugging in Gdb/Eclipse, I find that inside an object, I call a method of the same class, but suddently my this become 0.

I've found that not uncaching my pointers (adding 0x40000000) stop the crash. I really do not understand why at this point. And it seems to always be centered around the same piece of code.

Any pointer (no pun intended :) )?

[TyRaNiD]

Well in general you probably shouldn't use uncached addresses except in very specific circumstances.

The only thing I can think of is some data you are using it being written out at the cached address and read out at the uncached address, you cannot assume that the cache will ever be flushed before you use the real data, and the CPU is too stupid (or I suppose too trusting) to check whether the cache through address you are using is actually backed by cache. If it did it would kind of defeat the point :P

If that makes any sense what so ever :)

[Aion]

Yeah,

Once I noticed that I was using uncached data for more than the gu, I realized that I should stop doing so.

Anyway, I'm really really glad that it seems that I've found the root of the evil :P It was really driving me insane with the amount of wasted time trying to figure it out.

I'm not familiar enough with the underlying architecture to really understand why accessing the data directly instead of cached would create problems other than potential slowdown.

[TyRaNiD]

Well the problem is simple, if data exists in the cache it might not be actually in main memory so reading uncached you cannot be sure what you end up reading.