Okay, so I got winetest to do something rational, and now I'd like to see if it will work for someone else.
It's easy!
Just apply this patch to a current Wine tree: http://www.winehq.org/pipermail/wine-patches/2008-February/049675.html
Then configure and make Wine as you normally would. Then, cd programs/winetest ./runtests <your-tag-here> (The tag I used was jwhite-etch64; I think you should identify who you are, and then if you have multiple systems, your system within that).
If you have winetricks in your path, it should be completely unattended. (If you don't, you'll need to hang around to click 'go' to the Gecko install dialog).
After about 5 minutes, your test results should be run and sent up to test.winehq.org. The cron job runs every 5 minutes, so sometime after that, a reload of: firefox http://test.winehq.org/data/%60date +%Y%m%d`/
Should show you your results! Easy as Pi!
There is still quite a bit to do. First, I'm waiting for a 'real' winetest expert to wack me with a clue bat. But I'll stumble along until the hammer falls...
Second, I want to do a stupidity check and make sure that winetest essentially correlates to a make -k test run. If they are not really identical, then we need to fix that problem...
Next, I think we should probably shift the build-id from being the date to being the HEAD sha1, but that's a radical enough change that I felt I needed to start simpler and within the current conventions.
But I'd really appreciate it if folks could try this and see if we can gather some data quickly. In theory, we should be able to quickly see the common points of failure and know where to attack first.
Cheers,
Jeremy
Jeremy White wrote:
Okay, so I got winetest to do something rational, and now I'd like to see if it will work for someone else.
It's easy!
Just apply this patch to a current Wine tree: http://www.winehq.org/pipermail/wine-patches/2008-February/049675.html
Then configure and make Wine as you normally would. Then, cd programs/winetest ./runtests <your-tag-here> (The tag I used was jwhite-etch64; I think you should identify who you are, and then if you have multiple systems, your system within that).
If you have winetricks in your path, it should be completely unattended. (If you don't, you'll need to hang around to click 'go' to the Gecko install dialog).
After about 5 minutes, your test results should be run and sent up to test.winehq.org. The cron job runs every 5 minutes, so sometime after that, a reload of: firefox http://test.winehq.org/data/%60date +%Y%m%d`/
Should show you your results! Easy as Pi!
There is still quite a bit to do. First, I'm waiting for a 'real' winetest expert to wack me with a clue bat. But I'll stumble along until the hammer falls...
Second, I want to do a stupidity check and make sure that winetest essentially correlates to a make -k test run. If they are not really identical, then we need to fix that problem...
Next, I think we should probably shift the build-id from being the date to being the HEAD sha1, but that's a radical enough change that I felt I needed to start simpler and within the current conventions.
But I'd really appreciate it if folks could try this and see if we can gather some data quickly. In theory, we should be able to quickly see the common points of failure and know where to attack first.
Cheers,
Jeremy
Hi,
The good thing about having the winetest from one source(http://www.astro.gla.ac.uk/users/paulm/WRT/CrossBuilt/) is that all the output is actually from the same executable and it's easier to compare. If people are using different build-id we will end up with a lot of directories with single test in it.
Something we also have with the current tests is that the source files actually point to the correct version in CVS. I now see some strange links to refs on http://test.winehq.org/data/20080205/.
Even though the HEAD sha1 looks better than the date for a build-id it will again introduce a lot of directories with only a few tests in there.
I don't see a problem with running the winetest executable under Wine the same as I do under all kinds of Windows versions.
Just my $0.02
Hi Paul,
Something we also have with the current tests is that the source files actually point to the correct version in CVS. I now see some strange links to refs on http://test.winehq.org/data/20080205/.
Well, I'm afraid I'd argue that the current tests should be updated to point to git...
Even though the HEAD sha1 looks better than the date for a build-id it will again introduce a lot of directories with only a few tests in there.
Is that really true?
Alexandre tends to touch the tree once a day, usually all in one batch. If folks kick off winetest right before the leave for the day, wouldn't the odds be that everyone would have the same HEAD sha1? (I guess that requires people to run winetest from a clean, winehq only tree).
And even if it is true, isn't it more logically correct?
Right now, I could download a winetest.exe and use Wine-0.9.1 to run it, and it would be grouped in the same bucket as someone running off their own badly hacked wine tip tree. Doesn't that create misleading data?
However, I agree that if we changed that now it would be a mess; we'd need to change the script that presents the directories to group them by date so as to present the more recent sha1 keys.
I don't see a problem with running the winetest executable under Wine the same as I do under all kinds of Windows versions.
Perhaps you're right. Occam would suggest you are.
But it sticks in my craw. I have a source tree; I have built the files; I can run make test and it fails badly. I want to tell the world about my failures now (/me stamps his little feet <grin>).
Getting a .exe from you requires you to have reliably built winetest.exe, and does not allow me to test the tip *now*. Further, if there are any discrepancies introduced by the way my system builds the test .exe files compared to yours, running your winetest.exe won't catch them. (I realize that may be pure rationalization).
I think to some extent we are at different purposes, which may explain a reason to change.
I think winetest has long been used to test the tests themselves, and to identify the issues with the tests and fix them. That is, the variable is the tests, and the underlying OS (i.e. Windows) is essentially a constant.
But my purpose requires 2 variables - the tests changing as needed, but also the underlying OS - Wine - changing rapidly as well.
Cheers,
Jeremy
Jeremy White wrote:
Hi Paul,
Something we also have with the current tests is that the source files actually point to the correct version in CVS. I now see some strange links to refs on http://test.winehq.org/data/20080205/.
Well, I'm afraid I'd argue that the current tests should be updated to point to git...
And, putting my money where my mouth is, I've ostensibly fixed that now. The code supposedly can handle either git or cvs style revisions now (although detection is crude; if rev has a period, it's cvs :-/).
Cheers,
Jeremy
Hi Jeremy,
Jeremy White wrote:
Hi Paul,
Something we also have with the current tests is that the source files actually point to the correct version in CVS. I now see some strange links to refs on http://test.winehq.org/data/20080205/.
Well, I'm afraid I'd argue that the current tests should be updated to point to git...
Even though the HEAD sha1 looks better than the date for a build-id it will again introduce a lot of directories with only a few tests in there.
Is that really true?
Alexandre tends to touch the tree once a day, usually all in one batch. If folks kick off winetest right before the leave for the day, wouldn't the odds be that everyone would have the same HEAD sha1? (I guess that requires people to run winetest from a clean, winehq only tree).
And even if it is true, isn't it more logically correct?
Ok, so even if HEAD tends to be the same doesn't that mean the same test doesn't get run on Windows versions? With a common source (and I agree that there probably should be more) it better to compare.
If you build both the winetest your self and run the tests with that same winetest the objectivity goes away, not?
Right now, I could download a winetest.exe and use Wine-0.9.1 to run it, and it would be grouped in the same bucket as someone running off their own badly hacked wine tip tree. Doesn't that create misleading data?
That's true and I don't see an easy solution to that. There are loads of variables that influence the outcome of test as mentioned in this email but also in the rest of the email-chain. We already see failures with tests on Wine that were not present on AJ's box.
However, I agree that if we changed that now it would be a mess; we'd need to change the script that presents the directories to group them by date so as to present the more recent sha1 keys.
I don't see a problem with running the winetest executable under Wine the same as I do under all kinds of Windows versions.
Perhaps you're right. Occam would suggest you are.
But it sticks in my craw. I have a source tree; I have built the files; I can run make test and it fails badly. I want to tell the world about my failures now (/me stamps his little feet <grin>).
Getting a .exe from you requires you to have reliably built winetest.exe, and does not allow me to test the tip *now*. Further, if there are any discrepancies introduced by the way my system builds the test .exe files compared to yours, running your winetest.exe won't catch them. (I realize that may be pure rationalization).
'You' in this case is Paul Millar and not this Paul ;-).
I think to some extent we are at different purposes, which may explain a reason to change.
I think winetest has long been used to test the tests themselves, and to identify the issues with the tests and fix them. That is, the variable is the tests, and the underlying OS (i.e. Windows) is essentially a constant.
But my purpose requires 2 variables - the tests changing as needed, but also the underlying OS - Wine - changing rapidly as well.
Cheers,
Jeremy
On Feb 5, 2008 10:30 AM, Paul Vriens paul.vriens.wine@gmail.com wrote:
Hi,
The good thing about having the winetest from one source(http://www.astro.gla.ac.uk/users/paulm/WRT/CrossBuilt/) is that all the output is actually from the same executable and it's easier to compare. If people are using different build-id we will end up with a lot of directories with single test in it.
If some people went through the trouble of using buildid's and not the precompiled winetest, along with documenting their hardware/OS on a wiki page, for instance, then we could more information on where specific failures are occuring. For instance, I noticed Joapa was reporting some MSI bugs, and was the only one posting an MSI failure in the make test usually failing bug.
If some people went through the trouble of using buildid's and not the precompiled winetest, along with documenting their hardware/OS on a wiki page, for instance, then we could more information on where specific failures are occuring. For instance, I noticed Joapa was reporting some MSI bugs, and was the only one posting an MSI failure in the make test usually failing bug.
Actually, I think there is a better solution. If we want to capture additional system information, and there is a command line tool that would allow that, we can do so in the new 'runtests' script.
Right now, I have it capturing gcc, glibc, and kernel version info, and that is all readily available as part of the nifty winetest results page.
We could also conceivably add info on processor count and ram size, as well as video and audio card. (But those would require slightly more sophisticated scripts to postprocess; we don't want to just dump the results of xdpyinfo for example...).
Cheers,
Jeremy
On 05/02/2008, Jeremy White jwhite@codeweavers.com wrote:
If some people went through the trouble of using buildid's and not the precompiled winetest, along with documenting their hardware/OS on a wiki page, for instance, then we could more information on where specific failures are occuring. For instance, I noticed Joapa was reporting some MSI bugs, and was the only one posting an MSI failure in the make test usually failing bug.
Actually, I think there is a better solution. If we want to capture additional system information, and there is a command line tool that would allow that, we can do so in the new 'runtests' script.
For hardware and software information, capturing this and including it in the test results in an automated way is a good idea. Updates will change Windows and Wine behaviour and thus affect the results. It is not feasible to track this manually in a static page, nor reliable when looking back at the data.
That said, some general information may be useful along with the build id, for example:
id ; owner ; OS ; Machine Type (Laptop, Desktop, VM, ...) ; Description
Vista-HomePrem-NoUAC ; ReeceDunn ; Windows Vista Home Premium ; Notebook ; Same as Vista-HomePrem-UAC, but with UAC disabled. wine-0.9.54 ; ReeceDunn ; Ubuntu Gutsy Gibbon (7.10) ; Notebook ; Running with the Zune theme installed.
This would be supplementary to the automatically gathered data.
Another idea would be to store some user settings (like the above) for each build id. If the settings exist, use them, otherwise prompt the user for them. These can then be added into the Wine test results data.
Right now, I have it capturing gcc, glibc, and kernel version info, and that is all readily available as part of the nifty winetest results page.
We could also conceivably add info on processor count and ram size, as well as video and audio card. (But those would require slightly more sophisticated scripts to postprocess; we don't want to just dump the results of xdpyinfo for example...).
It would also be useful to detect driver name and version (e.g. nv, proprietry NVidia or nouveau) to help track regressions.
- Reece
-
Austin English -
Jeremy White -
Jeremy White -
Paul Vriens -
Reece Dunn