The buildbot does three things to get fast, reliable test runs: - it runs a 'safe' set of tests in the background, without DISPLAY set, in a different wineprefix; this saves three minutes - it has a blacklist of known flaky or broken tests, listing how it's broken and linking to the bug report - it kills wineserver after the run (since some tests leave processes around)
I hoisted these out of the buildbot config and into a separate script, http://code.google.com/p/winezeug/source/browse/trunk/buildbot/dotests.sh
As a bonus, you can ask it to run all the flaky tests instead of all the good ones, which might be useful if you're trolling for test bugs to fix.
Here's the current blacklist from that script:
# Blacklist format # test condition [bug] # Tests may appear multiple times in the list. # Supported conditions: # FLAKY - fails sometimes # CRASHY - crashes sometimes # SYS - always fails on some systems, not on others # HEAP - fails or crashes if warn+heap # NOTTY - test fails if output redirected # BAD64 - always fails on 64 bits
cat > /tmp/blacklist.txt <<_EOF_ dlls/kernel32/tests/heap.ok HEAP dlls/dsound/tests/ds3d8.ok SYS 8668 dlls/dsound/tests/ds3d.ok SYS 8668 dlls/dsound/tests/dsound8.ok SYS 8668 dlls/dsound/tests/dsound.ok SYS 8668 dlls/user32/tests/input.ok SYS 12053 dlls/user32/tests/msg.ok SYS 12053 dlls/user32/tests/win.ok SYS 12053 dlls/oleaut32/tests/tmarshal.ok BAD64 26768 dlls/ieframe/tests/ie.ok BAD64 26768 dlls/ws2_32/tests/sock.ok CRASHY 28012 dlls/wininet/tests/urlcache.ok FLAKY 28038 dlls/winmm/tests/wave.ok SYS 28048 dlls/winmm/tests/mci.ok SYS 28071 dlls/urlmon/tests/url.ok FLAKY 28108 dlls/winmm/tests/capture.ok SYS 28109 dlls/shell32/tests/shlfolder.ok SYS 28216 dlls/kernel32/tests/process.ok NOTTY 28220 dlls/d3dx9_36/tests/shader.ok HEAP 28255 dlls/kernel32/tests/pipe.ok HEAP 28257 programs/cmd/tests/batch.ok HEAP 28258 dlls/dsound/tests/ds3d8.ok HEAP 28260 dlls/dsound/tests/ds3d.ok HEAP 28260 dlls/mshtml/tests/style.ok HEAP 28262 dlls/winhttp/tests/winhttp.ok SYS 28267 programs/wscript/tests/run.ok BAD64 28285 dlls/msctf/tests/inputprocessor.ok FLAKY 28288 _EOF_
On Tue, 6 Sep 2011, Dan Kegel wrote:
The buildbot does three things to get fast, reliable test runs:
[...]
- it kills wineserver after the run (since some tests leave processes around)
I found that most of the processes I had that were left behind were winedbg processes showing the crash dialog. This was slowing things down quite a bit with winetest because each was triggering the two minutes timeout.
I know you're avoiding crashing tests and you probably already avoid this dialog but here goes anyway; I prime all my fresh wineprefixes with:
./wine "./programs/regedit/regedit.exe.so" - <<__EOF__ [HKEY_CURRENT_USER\Software\Wine\WineDbg] "ShowCrashDialog"=dword:00000000 __EOF__
And now I'll plug the conformance tests Wiki page where I documented this: http://wiki.winehq.org/ConformanceTests
On Tue, Sep 6, 2011 at 2:12 AM, Francois Gouget fgouget@free.fr wrote:
I found that most of the processes I had that were left behind were winedbg processes showing the crash dialog. This was slowing things down quite a bit with winetest because each was triggering the two minutes timeout.
In my case, it is iexplore -Embedding and rpcss, and only on win64?
I know you're avoiding crashing tests and you probably already avoid this dialog but here goes anyway; I prime all my fresh wineprefixes with:
./wine "./programs/regedit/regedit.exe.so" - <<__EOF__ [HKEY_CURRENT_USER\Software\Wine\WineDbg] "ShowCrashDialog"=dword:00000000 __EOF__
I did that, but then I hit crashes that broke wineboot, so I switched to applying http://code.google.com/p/winezeug/source/browse/trunk/buildbot/winedbg-ignor... which bakes that behavior in as default.
Another annoyance: winedbg is hanging for me on exit after doing the backtrace, http://bugs.winehq.org/show_bug.cgi?id=28265
And now I'll plug the conformance tests Wiki page where I documented this: http://wiki.winehq.org/ConformanceTests
A fine page. - Dan
-
Dan Kegel -
Francois Gouget