[Bug 58698] New: Application goes into an infinite loop under new wow64 but works okay under old wow64

[WineHQ Bugzilla]

http://bugs.winehq.org/show_bug.cgi?id=58698

Bug ID: 58698
           Summary: Application goes into an infinite loop under new wow64
                    but works okay under old wow64
           Product: Wine
           Version: 10.15
          Hardware: x86-64
                OS: Linux
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: ntdll
          Assignee: [e-mail]
          Reporter: [e-mail]
      Distribution: ---

Created attachment 79297
  --> http://bugs.winehq.org/attachment.cgi?id=79297
WINEDEBUG=+virtual under "new wow64"

Application goes into an infinite loop under new wow64 but works okay under old
wow64

I have an application that enters an infinite loop under "new wow64" but works
okay with "old wow64". Running under actual 64-bit Windows is okay too. The
problem seems to be caused by application logic which calls VirtualAlloc()
repeatedly in order to (presumably) find out the maximum available amount of
memory that the application is able to allocate.

The first hint that this might be somehow related to memory were the error
messages reported when running under "old wow64" :

01a0:err:virtual:map_view anon mmap error Cannot allocate memory, size
0x80010000, unix_prot 0
01a0:err:virtual:allocate_virtual_memory out of memory for allocation, base
(nil) size 80000000
01a0:err:virtual:map_view anon mmap error Cannot allocate memory, size
0x78010000, unix_prot 0
01a0:err:virtual:allocate_virtual_memory out of memory for allocation, base
(nil) size 78000000
01a0:err:virtual:map_view anon mmap error Cannot allocate memory, size
0x74010000, unix_prot 0
01a0:err:virtual:allocate_virtual_memory out of memory for allocation, base
(nil) size 74000000

Running the application with WINEDEBUG=+virtual showed the
NtAllocateVirtualMemory calls that are made by the application, starting with a
size of 0x7fffffff, exploring a number of different sizes, and then eventually
settling on 0x701f0000 :

01a0:trace:virtual:NtAllocateVirtualMemory 0xffffffff (nil) 7fffffff 2000
00000004
01a0:err:virtual:map_view anon mmap error Cannot allocate memory, size
0x80010000, unix_prot 0
01a0:err:virtual:allocate_virtual_memory out of memory for allocation, base
(nil) size 80000000
01a0:trace:virtual:NtFreeVirtualMemory 0xffffffff (nil) 00000000 8000
01a0:trace:virtual:NtAllocateVirtualMemory 0xffffffff (nil) 3fffffff 2000
00000004
01a0:trace:virtual:map_view got mem in reserved area 0x11050000-0x51050000
01a0:trace:virtual:dump_view View: 0x11050000 - 0x5104ffff (valloc)
01a0:trace:virtual:dump_view       0x11050000 - 0x5104ffff --rw-
01a0:trace:virtual:NtFreeVirtualMemory 0xffffffff 0x11050000 00000000 8000
-- snip : successful calls after that with sizes 0x5fffffff , 0x6fffffff --
01a0:trace:virtual:NtAllocateVirtualMemory 0xffffffff (nil) 77ffffff 2000
00000004
01a0:err:virtual:map_view anon mmap error Cannot allocate memory, size
0x78010000, unix_prot 0
01a0:err:virtual:allocate_virtual_memory out of memory for allocation, base
(nil) size 78000000
01a0:trace:virtual:NtFreeVirtualMemory 0xffffffff (nil) 00000000 8000
-- snip : goes on to try sizes starting with 0x70 ... --
01a0:trace:virtual:NtAllocateVirtualMemory 0xffffffff (nil) 701f0001 2000
00000004
01a0:err:virtual:map_view anon mmap error Cannot allocate memory, size
0x70201000, unix_prot 0
01a0:err:virtual:allocate_virtual_memory out of memory for allocation, base
(nil) size 701f1000
01a0:trace:virtual:NtFreeVirtualMemory 0xffffffff (nil) 00000000 8000
01a0:trace:virtual:NtAllocateVirtualMemory 0xffffffff (nil) 701f0000 2000
00000004
01a0:trace:virtual:map_view got mem with anon mmap 0x80000000-0xf01f0000
01a0:trace:virtual:dump_view View: 0x80000000 - 0xf01effff (valloc)
01a0:trace:virtual:dump_view       0x80000000 - 0xf01effff --rw-
01a0:trace:virtual:NtFreeVirtualMemory 0xffffffff (nil) 00000000 8000

The application makes the same calls under "new wow64", but since none of those
calls ever fail, it seems to get confused, reaches size=0, and then gets stuck
on that size :

016c:trace:virtual:NtAllocateVirtualMemory 0xffffffffffffffff (nil) 7fffffff
2000 00000004
016c:trace:virtual:map_view got mem with map_free_area 0x7fff0000-0xffff0000
016c:trace:virtual:dump_view View: 0x7fff0000 - 0xfffeffff (valloc)
016c:trace:virtual:dump_view       0x7fff0000 - 0xfffeffff --rw-
016c:trace:virtual:NtFreeVirtualMemory 0xffffffffffffffff 0x7fff0000 00000000
8000
016c:trace:virtual:NtAllocateVirtualMemory 0xffffffffffffffff (nil) 3fffffff
2000 00000004
016c:trace:virtual:map_view got mem in reserved area 0x11050000-0x51050000
016c:trace:virtual:dump_view View: 0x11050000 - 0x5104ffff (valloc)
016c:trace:virtual:dump_view       0x11050000 - 0x5104ffff --rw-
016c:trace:virtual:NtFreeVirtualMemory 0xffffffffffffffff 0x11050000 00000000
8000
016c:trace:virtual:NtAllocateVirtualMemory 0xffffffffffffffff (nil) 1fffffff
2000 00000004
016c:trace:virtual:map_view got mem in reserved area 0x11050000-0x31050000
016c:trace:virtual:dump_view View: 0x11050000 - 0x3104ffff (valloc)
016c:trace:virtual:dump_view       0x11050000 - 0x3104ffff --rw-
016c:trace:virtual:NtFreeVirtualMemory 0xffffffffffffffff 0x11050000 00000000
8000
016c:trace:virtual:NtAllocateVirtualMemory 0xffffffffffffffff (nil) 0fffffff
2000 00000004
016c:trace:virtual:map_view got mem in reserved area 0x11050000-0x21050000
016c:trace:virtual:dump_view View: 0x11050000 - 0x2104ffff (valloc)
016c:trace:virtual:dump_view       0x11050000 - 0x2104ffff --rw-
-- snip : repeated calls with size right-shifted by 1 --
016c:trace:virtual:NtAllocateVirtualMemory 0xffffffffffffffff (nil) 0000000f
2000 00000004
016c:trace:virtual:map_view got mem in reserved area 0x3750000-0x3751000
016c:trace:virtual:dump_view View: 0x3750000 - 0x3750fff (valloc)
016c:trace:virtual:dump_view       0x3750000 - 0x3750fff --rw-
016c:trace:virtual:NtFreeVirtualMemory 0xffffffffffffffff 0x3750000 00000000
8000
016c:trace:virtual:NtAllocateVirtualMemory 0xffffffffffffffff (nil) 00000007
2000 00000004
016c:trace:virtual:map_view got mem in reserved area 0x3750000-0x3751000
016c:trace:virtual:dump_view View: 0x3750000 - 0x3750fff (valloc)
016c:trace:virtual:dump_view       0x3750000 - 0x3750fff --rw-
016c:trace:virtual:NtFreeVirtualMemory 0xffffffffffffffff 0x3750000 00000000
8000
016c:trace:virtual:NtAllocateVirtualMemory 0xffffffffffffffff (nil) 00000003
2000 00000004
016c:trace:virtual:map_view got mem in reserved area 0x3750000-0x3751000
016c:trace:virtual:dump_view View: 0x3750000 - 0x3750fff (valloc)
016c:trace:virtual:dump_view       0x3750000 - 0x3750fff --rw-
016c:trace:virtual:NtFreeVirtualMemory 0xffffffffffffffff 0x3750000 00000000
8000
016c:trace:virtual:NtAllocateVirtualMemory 0xffffffffffffffff (nil) 00000001
2000 00000004
016c:trace:virtual:map_view got mem in reserved area 0x3750000-0x3751000
016c:trace:virtual:dump_view View: 0x3750000 - 0x3750fff (valloc)
016c:trace:virtual:dump_view       0x3750000 - 0x3750fff --rw-
016c:trace:virtual:NtFreeVirtualMemory 0xffffffffffffffff 0x3750000 00000000
8000
016c:trace:virtual:NtAllocateVirtualMemory 0xffffffffffffffff (nil) 00000000
2000 00000004
016c:trace:virtual:NtFreeVirtualMemory 0xffffffffffffffff (nil) 00000000 8000
016c:trace:virtual:NtAllocateVirtualMemory 0xffffffffffffffff (nil) 00000000
2000 00000004
016c:trace:virtual:NtFreeVirtualMemory 0xffffffffffffffff (nil) 00000000 8000
016c:trace:virtual:NtAllocateVirtualMemory 0xffffffffffffffff (nil) 00000000
2000 00000004
016c:trace:virtual:NtFreeVirtualMemory 0xffffffffffffffff (nil) 00000000 8000

It then loops on the call with size=0 indefinitely and keeps running until I
kill the wineserver.

Unfortunately, this is a proprietary legacy application so I can't share it,
however I'll be more than glad to accept pointers on which debug channels to
activate, any other further debugging steps, or changes to the code that might
fix this.

Thanks in advance.

[Bug 58698] Application goes into an infinite loop under new wow64 but works okay under old wow64

[WineHQ Bugzilla]

http://bugs.winehq.org/show_bug.cgi?id=58698

--- Comment #1 from Daniel Kamil Kozar  ---
Created attachment 79298
  --> http://bugs.winehq.org/attachment.cgi?id=79298
WINEDEBUG=+virtual under "old wow64"

[Bug 58698] Application goes into an infinite loop under new wow64 but works okay under old wow64

[WineHQ Bugzilla]

http://bugs.winehq.org/show_bug.cgi?id=58698

LingM  changed:

What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |[e-mail]

[Bug 58698] Application goes into an infinite loop under new wow64 but works okay under old wow64

[WineHQ Bugzilla]

http://bugs.winehq.org/show_bug.cgi?id=58698

--- Comment #2 from Daniel Kamil Kozar  ---
After some more peeking and poking, I was able to pinpoint the exact point
where the infinite loop happens. Attached is a disassembly with some comments
inline.

I understand that code like this isn't very common as otherwise this would've
already been reported. Additionally, making the code jump out of the loop on
VirtualAlloc's first success (so, basically replacing "je .fail" with "jne
.epilogue" and NOPing things out to make sense) makes everything work properly,
which shows that this code doesn't seem to have a real purpose and the rest of
the application doesn't seem to care or is able to adapt to an allocation being
smaller than "all available memory".

64-bit Windows behaviour when running 32-bit applications seems to follow "old
WoW64" behaviour though : a short test shows that calling VirtualAlloc(0,
0x7fffffff,0x2000,4) does indeed return zero, which makes this code work even
though it's flawed.

Perhaps it's worth considering adding a registry flag or an environment
variable for a "maximum VirtualAlloc allocation size" in new WoW64 to combat
cases like these.

[Bug 58698] Application goes into an infinite loop under new wow64 but works okay under old wow64

[WineHQ Bugzilla]

http://bugs.winehq.org/show_bug.cgi?id=58698

--- Comment #3 from Daniel Kamil Kozar  ---
Created attachment 79381
  --> http://bugs.winehq.org/attachment.cgi?id=79381
Disassembly of the function calling VirtualAlloc