Hi,
Erich Hoover wrote: >so if we're not testing
the version we wouldn't know that it got removed.
This is a very valid point. For instance, I'm currently writing a test that will read ok(1234123123==mhdr.dwOffset || broken(0==mhdr.dwOffset/*w9x,nt*/) ...) i.e. w2k+xp+Vista+7 differ from w95+98+me+nt. The Wine testsuite will not warn should MS-Windows ever revert back to w9x+nt behaviour.
I've been fixing multimedia bugs in Wine, some of which were present for almost 10 years now. My point is: right *now* I'm closely looking at the exact behaviour under existing versions of MS-Windows (and having Wine mimick "modern" w2k/xp behaviour), yet the broken() mechanism prevents the tests from noticing when that behaviour will change next time.
Patches these days make Wine implement the now "modern" behaviour, but remember modern is relative and tests with broken() will fail to report any future change... Until, perhaps again in ten years, somebody will look closely at the results, perhaps triggered by a bug report. Why wait for a bug report?
Somehow, I feel that not noticing changes in behaviour is a waste of the test automation resources.
broken() shadows broken (not valid any more) assumptions.
Every now and then I wish there were a switch to disable broken() entirely and see what test.winehq reports for the various OS. E.g. $WINETEST_STRICT=1 => have broken() always yield 0/false.
The goal to reach becomes WINETEST_STRICT=1 => no test failures on the system Wine purports to mimick (currently XP(?)). (Seems like a good idea, expect a patch soon ;)
Regards, Jörg Höhle Well, at least the above example of broken() is much stricter than ok(!rc||broken(rc/*w9x*/)) == anything goes What does such a test tell except impose Wine's rc?