Extended test suite

[Giovanni Mascellani]

Hi,

some time ago I was asked by a reviewer to modify a MR I had submitted 
to remove some of the conformance tests I had added with it. The 
concern, which I find completely valid, was that since the tests are run 
on each MR submission we should try to reduce (or, at least, do not 
increase too much) the run time, otherwise it becomes too much 
inconvenient for everybody.

As I said, that totally makes sense for me, and in that particular 
occasion I ended up removing some cases, hopefully keeping enough to 
cover all the common test cases. But corner cases exist too, and they 
can make a developer waste a lot of time chasing them down. And at least 
in my experience it's often hard to predict which specific cases are 
going to be somehow critical in the future. So giving up some tests 
bears most of the times if not always the risk of having to waste time 
on something that would have been simpler if only we had kept that 
specific test case.

So I'm curious to know whether anybody else experiences this problem, 
and possibly brainstorm some ideas about how to work it around.

What I'd like to propose is to introduce an extended test suite, which 
in practice means adding a flag to our current test suite and gating 
some tests to be executed (at all, or to be executed on a larger number 
of cases) only if the flag is set.

The regular test suite could be run on all MRs, and the extended test 
suite could be run perhaps once per day or so. The feedback is less 
immediate, but it still looks better than dropping those tests at all.

The terms "pre-merge test" and "post-merge tests" are sometimes used, I 
think they essentially mean the same thing.

Is there any interest for this? Or alternative solutions?

Gio.

[Gabriel Ivăncescu]

Hi,

For what it's worth, I'm personally in favor of this, if people need it.
It could be a mode like WINETEST_INTERACTIVE=1 but for extended tests.

The problem, of course, is that initially you won't have many users of
this in the test suite so it's hard to justify it. And it won't have
until it's available. The classic chicken-and-egg problem.