Generating stats on regression bugs in bugzilla

[Scott Ritchie]

I had a theory that we might be getting better at preventing regressions 
now than a year or so ago due to the expansion of things like AppDB and 
the test suite.  It'd be nice to have some sort of data though.

Would it be reasonable to comb through the bugzilla database for all the 
bugs tagged regression and analyze them?  I'm interested in:

1) quantity and frequency of bugs tagged regression that were filed at 
various dates since the release of 1.0.  I'm not sure whether it's 
better to use the version number field in bugzilla as a proxy for "first 
version with regression" or to just guess that from the date.
2) how long regression bugs took to be fixed

Thanks,
Scott Ritchie

[Austin English]

On Fri, Apr 24, 2009 at 6:40 PM, Scott Ritchie  wrote:
> I had a theory that we might be getting better at preventing regressions now
> than a year or so ago due to the expansion of things like AppDB and the test
> suite.  It'd be nice to have some sort of data though.
>
> Would it be reasonable to comb through the bugzilla database for all the
> bugs tagged regression and analyze them?  I'm interested in:
> - quantity and frequency of bugs tagged regression that were filed at
>
> various dates since the release of 1.0.  I'm not sure whether it's better to
> use the version number field in bugzilla as a proxy for "first version with
> regression" or to just guess that from the date.
> 2) how long regression bugs took to be fixed
>
> Thanks,
> Scott Ritchie

It'd be better to use the SHA1SUM of the bad patch, which will give
you the day the regression was added.

If the fix has the SHA1SUM listed in there as well, it's even easier for you.

[Ken Sharp]

Austin English wrote:
> On Fri, Apr 24, 2009 at 6:40 PM, Scott Ritchie  wrote:
> > I had a theory that we might be getting better at preventing regressions now
> > than a year or so ago due to the expansion of things like AppDB and the test
> > suite.  It'd be nice to have some sort of data though.
> >
> > Would it be reasonable to comb through the bugzilla database for all the
> > bugs tagged regression and analyze them?  I'm interested in:
> > - quantity and frequency of bugs tagged regression that were filed at
> >
> > various dates since the release of 1.0.  I'm not sure whether it's better to
> > use the version number field in bugzilla as a proxy for "first version with
> > regression" or to just guess that from the date.
> > 2) how long regression bugs took to be fixed
> >
> > Thanks,
> > Scott Ritchie
>
> It'd be better to use the SHA1SUM of the bad patch, which will give
> you the day the regression was added.
>
> If the fix has the SHA1SUM listed in there as well, it's even easier for you.

Some bugs contain the wrong regression before mentioning the correct 
one.  It would be necessary to look for the last SHA1SUM mentioned in a 
bug to get the correct one.

But if the last comment is the fix... you see what I'm saying.

[Austin English]

On Fri, Apr 24, 2009 at 8:04 PM, Ken Sharp  wrote:
> Austin English wrote:
> > On Fri, Apr 24, 2009 at 6:40 PM, Scott Ritchie 
> > wrote:
> > > I had a theory that we might be getting better at preventing regressions
> > > now
> > > than a year or so ago due to the expansion of things like AppDB and the
> > > test
> > > suite.  It'd be nice to have some sort of data though.
> > >
> > > Would it be reasonable to comb through the bugzilla database for all the
> > > bugs tagged regression and analyze them?  I'm interested in:
> > > - quantity and frequency of bugs tagged regression that were filed at
> > >
> > > various dates since the release of 1.0.  I'm not sure whether it's better
> > > to
> > > use the version number field in bugzilla as a proxy for "first version
> > > with
> > > regression" or to just guess that from the date.
> > > 2) how long regression bugs took to be fixed
> > >
> > > Thanks,
> > > Scott Ritchie
> >
> > It'd be better to use the SHA1SUM of the bad patch, which will give
> > you the day the regression was added.
> >
> > If the fix has the SHA1SUM listed in there as well, it's even easier for
> > you.
>
> Some bugs contain the wrong regression before mentioning the correct one.
>  It would be necessary to look for the last SHA1SUM mentioned in a bug to
> get the correct one.
>
> But if the last comment is the fix... you see what I'm saying.

You'll also want to account for when a regression is reported/fixed.
E.g., some regressions were introduced in, say, 1.1.12, but not
reported until 1.1.15.

And if the fix is put in immediately, but not marked fixed until
1.1.18, how would we know? Without the SHA1SUM, it's hard to say.

Not trying to discourage you, as it would be _very_ interesting to
see, but I don't think it'll be very easy at all.