http://bugs.winehq.org/show_bug.cgi?id=58698
--- Comment #2 from Daniel Kamil Kozar dkk089@gmail.com --- After some more peeking and poking, I was able to pinpoint the exact point where the infinite loop happens. Attached is a disassembly with some comments inline.
I understand that code like this isn't very common as otherwise this would've already been reported. Additionally, making the code jump out of the loop on VirtualAlloc's first success (so, basically replacing "je .fail" with "jne .epilogue" and NOPing things out to make sense) makes everything work properly, which shows that this code doesn't seem to have a real purpose and the rest of the application doesn't seem to care or is able to adapt to an allocation being smaller than "all available memory".
64-bit Windows behaviour when running 32-bit applications seems to follow "old WoW64" behaviour though : a short test shows that calling VirtualAlloc(0, 0x7fffffff,0x2000,4) does indeed return zero, which makes this code work even though it's flawed.
Perhaps it's worth considering adding a registry flag or an environment variable for a "maximum VirtualAlloc allocation size" in new WoW64 to combat cases like these.