On Sun, 28 Oct 2007, Reece Dunn wrote: [...]
It would be even better if the tests were also run on real machines, as that would catch which test failures are VM related (such as the Direct3D tests).
Sure. However I don't have real Windows machines so I'll leave this as an exercise for someone else. The tricky part is scripting the download, signature checking and getting it all to run automatically at night (because if it's not 100% automated I don't think we'll have regular results). Maybe the way to go is with the Windows scheduler plus a wsh script. Or winetest could be modified to do most of it: with wininet/winhttp/urlmon managing the download part should be quite feasible, while crypt32 might provide for the signature checking part. Then it's just a matter for winetest to remain idle (with a systray icon?) until the specified time to run the tests...
Actually, I'll mention that Virtual machines do have an advantage over real Windows machine: after the tests are run, they are reset back to a known state, even if the tests did not clean things up quite right.