Hi,
some time ago I was asked by a reviewer to modify a MR I had submitted
to remove some of the conformance tests I had added with it. The
concern, which I find completely valid, was that since the tests are run
on each MR submission we should try to reduce (or, at least, do not
increase too much) the run time, otherwise it becomes too much
inconvenient for everybody.
As I said, that totally makes sense for me, and in that particular
occasion I ended up removing some cases, hopefully keeping enough to
cover all the common test cases. But corner cases exist too, and they
can make a developer waste a lot of time chasing them down. And at least
in my experience it's often hard to predict which specific cases are
going to be somehow critical in the future. So giving up some tests
bears most of the times if not always the risk of having to waste time
on something that would have been simpler if only we had kept that
specific test case.
So I'm curious to know whether anybody else experiences this problem,
and possibly brainstorm some ideas about how to work it around.
What I'd like to propose is to introduce an extended test suite, which
in practice means adding a flag to our current test suite and gating
some tests to be executed (at all, or to be executed on a larger number
of cases) only if the flag is set.
The regular test suite could be run on all MRs, and the extended test
suite could be run perhaps once per day or so. The feedback is less
immediate, but it still looks better than dropping those tests at all.
The terms "pre-merge test" and "post-merge tests" are sometimes used, I
think they essentially mean the same thing.
Is there any interest for this? Or alternative solutions?
Gio.