Fixing make test - progress update

[Jeremy White]

I believe that we have an easy way
to find out the 'state of the art' wrt to make test.

That is, my patch + request of a few days ago:
  http://www.winehq.org/pipermail/wine-devel/2008-February/062303.html
resulted in a very nice, easy to digest report
  http://test.winehq.org/data/20080205/

That data has five test runs and shows clear across
the board failures in msi:install, shell32:shellink,
user32:listbox, and user32:msg.

So, we can quickly and easily collate and visually see
the make test failures.

And, heck, I'm sure that fixing those failures is now
a trivial exercise.

Now we just need the will.  :-/.

Not sure how to arrange that, but I think having make
test succeed on all systems should be a mandatory
requirement for Wine 1.0, so I think we have to work
on solving it sooner rather than later.

The todo's as I see them are as follows:

1.  Revise winetest to not require the 'make dist'
      step, so that the function of my patch can
      be implemented more cleanly.

2.  Persuade people to run a script on a regular
      basis so we always have this data.  It's
      trickier than the average bear because the tests
      require the 'real' console, so you can't just
      stick it in a headless cron job.  Maybe some
      xauth magic... :-/

3.  (The hard one)  Focus in and fix the issues.

I'll work on #1 and #2 (although not for a week or two),
if everyone else agrees to do #3 <evil grin>.

Cheers,

Jeremy

[Paul Vriens]

Jeremy White wrote:
> I believe that we have an easy way
> to find out the 'state of the art' wrt to make test.
>
> That is, my patch + request of a few days ago:
>   http://www.winehq.org/pipermail/wine-devel/2008-February/062303.html
> resulted in a very nice, easy to digest report
>   http://test.winehq.org/data/20080205/

Hi Jeremy,

I don't want to spoil your enthusiasm but how does that differ from for example 
(old one though) http://test.winehq.org/data/200801301937/#Wine ?

[Jeremy White]

> I don't want to spoil your enthusiasm but how does that differ from for example 
> (old one though) http://test.winehq.org/data/200801301937/#Wine ?

The main difference is that I ran it from my Wine tree, and getting it built
and ready to go was completely under my control.  That is, this is as close
to running make test in my tree, but dumping the results to a web site
for collation.

There is an arguably theoretical difference in that it was built with gcc
and my tool chain and is more theoretically identical to a 'make test' run
than running a prebuilt winetest.exe.

Cheers,

Jeremy

[Reece Dunn]

On 08/02/2008, Jeremy White  wrote:
> That data has five test runs and shows clear across
> the board failures in msi:install, shell32:shellink,
> user32:listbox, and user32:msg.

The user32:listbox tests fail everywhere when not run from the
dlls/user32/tests directory! I have a patch in wine-patches that is in
current git (http://source.winehq.org/git/wine.git/?a=commitdiff;h=01af8501b992896a5fff38...)
 that fixes those failures.

I had a look at the user32:msg failures, but the test results for
these on Wine appear to be completely different on different wine
versions. I suspect these are down to configuration variations (e.g. X
vs. Wine window handling (with focus behaviour) and the like).

I haven't had a look at the others yet.

- Reece