Tale of two feedbacks
We have started pushing code coverage results and associated analysis published at http://wiki.mozilla.org/Firefly to mozilla community in the form of brown bags and posts on dev.platform & dev.planning.
I want to share two success stories with you so that it inspires you to create more of the same.
Success story 1: Heros : Robert Strong & Ed@Siliconforks.com
Robert looked at the javascript code coverage results and immediately found an issue. He has made spcific code changes for 'xpcom-shutdown' case-handling in nsUpdateService.js and he did not see any coverage data on the same. Robert has test cases to cover that area and they are getting run and passing....
A bug is filed to track the issue on " Testing - > code coverage " . Robert and I debugeed the code and found that there should be an error in the way the coverage data is getting collected. There should be a bug in the coverage tool itself.
So, I contacted Ed who is the main developer on the jscoverage tool [ that we are using for javascript code coverage report generation ] with the bug details.
With in 24 hours, Ed found and fixed the problem in jscoverage !
Now we have a happy customer, a better code coverage report for jscoverage [ from the next run obviously ] and a win win for mozilla community as well as jscoverage users at large.
Moral of the story : Robert's willingness to look at javascript code coverage reports helped fix a bug in the coverage tool which resulted in mutual benefit for Mozilla and JavaScript code coverage community world wide.
Success story 2: Heros : Dave Townsend
In the C/C++ coverage arena, the coverage numbers for xpinstall are suspiciously low. Dave Townsend looked at the data and reported back to me that this should not be the case since we have tests to cover that area.
So, we filed a Bug on " Testing -> code coverage " and started debugging the issue.
It turns out that in Instrumented build mode , we are able to run only 109 of 195 tests in the browser-chrome suite and the tests that are not getting run are related to xpinstall.
It also turns out that while we don't see this issue with nightly buildbot runs, we see this only on the instrumented build.
We have also figured out that when the browser-chrome tests are getting run, Giga Bytes of logs are getting generated. We have a 6.8 GB of Javascript errors while running 195 tests. Compared to 72000+ tests in mochitest suite we get a log of few MB.
These bloated logs started happening after the June 29th code coverage run. So, some thing is not right some where related to bloated logs.
Also , the build system is force killing the browser-chrome run after 42000 seconds of inactivity. Which means there is a test which is hanging the system at around 109th test and killing the rest of the job.
The issue is still not resolved and we are investigating the matter. Any help from community is very much appreciated.
Moral of the story: We would not have found this issue , had Dave not taken interest in the coverage results. You can all do the same and help make the system better.
I want to share two success stories with you so that it inspires you to create more of the same.
Success story 1: Heros : Robert Strong & Ed@Siliconforks.com
Robert looked at the javascript code coverage results and immediately found an issue. He has made spcific code changes for 'xpcom-shutdown' case-handling in nsUpdateService.js and he did not see any coverage data on the same. Robert has test cases to cover that area and they are getting run and passing....
A bug is filed to track the issue on " Testing - > code coverage " . Robert and I debugeed the code and found that there should be an error in the way the coverage data is getting collected. There should be a bug in the coverage tool itself.
So, I contacted Ed who is the main developer on the jscoverage tool [ that we are using for javascript code coverage report generation ] with the bug details.
With in 24 hours, Ed found and fixed the problem in jscoverage !
Now we have a happy customer, a better code coverage report for jscoverage [ from the next run obviously ] and a win win for mozilla community as well as jscoverage users at large.
Moral of the story : Robert's willingness to look at javascript code coverage reports helped fix a bug in the coverage tool which resulted in mutual benefit for Mozilla and JavaScript code coverage community world wide.
Success story 2: Heros : Dave Townsend
In the C/C++ coverage arena, the coverage numbers for xpinstall are suspiciously low. Dave Townsend looked at the data and reported back to me that this should not be the case since we have tests to cover that area.
So, we filed a Bug on " Testing -> code coverage " and started debugging the issue.
It turns out that in Instrumented build mode , we are able to run only 109 of 195 tests in the browser-chrome suite and the tests that are not getting run are related to xpinstall.
It also turns out that while we don't see this issue with nightly buildbot runs, we see this only on the instrumented build.
We have also figured out that when the browser-chrome tests are getting run, Giga Bytes of logs are getting generated. We have a 6.8 GB of Javascript errors while running 195 tests. Compared to 72000+ tests in mochitest suite we get a log of few MB.
These bloated logs started happening after the June 29th code coverage run. So, some thing is not right some where related to bloated logs.
Also , the build system is force killing the browser-chrome run after 42000 seconds of inactivity. Which means there is a test which is hanging the system at around 109th test and killing the rest of the job.
The issue is still not resolved and we are investigating the matter. Any help from community is very much appreciated.
Moral of the story: We would not have found this issue , had Dave not taken interest in the coverage results. You can all do the same and help make the system better.
Comments