Under the Hood :: CodeCoverage Infrastructure
Our goal is to provide weekly code coverage results for C/C++ and Javascript code portions of Firefox.
This regular reporting of code coverage should help development leads and developers to assess the risk factors associated with coverage gaps and help the community to get involved in targeted test development exercises.
However, it is beneficial to know the assembly line dependencies for code coverage data generation exercise.
There are four major parts in the code coverage results generation.
Almost always, the build infrastructure can be ruled out as the cause of failure. If that were the case, the entire build setup would catch FIRE. So, if the results are not there, please call QA points of contact who are Murali and Tim who know the reasons for failure.
Many times, the reason for failure happens to be the way we are picking up bleeding edge nightly trunk code base to create the executable. Creating an instrumented executable adds a lot more sensitivity to the product compared to an optimized build. One way, it is good that we are able to find some bugs that could not have been found with regular optimized builds.
In order to improve the infrastructure resiliency, we are making continuous improvements to the way we run tests. Recently, we created serialized Mochitest target which runs the big 72K+ tests in smaller chunks serially so that even if there is a crash/hang of Firefox during test run, it is localized only to the smaller chunk.
In few more days, if you are building and running tests on a regular nightly executable [ as a community volunteer ] you can use 'make mochitest-serial' to run Mochitests in smaller batches instead of your current option 'make mochitest-plain'.
Coverage reports are provided regularly at the following locations.
Thank you.
This regular reporting of code coverage should help development leads and developers to assess the risk factors associated with coverage gaps and help the community to get involved in targeted test development exercises.
However, it is beneficial to know the assembly line dependencies for code coverage data generation exercise.
There are four major parts in the code coverage results generation.
- The Instrumented executable of Firefox [ which is usually created off of trunk ]
- Various automated test suites that run on top of the instrumented build [ Xpcshell, reftests,crashtests, Mochitest, A11y, Chrome , Browser chrome etc., ]
- The codecoverage build factory in the buildbot that executes the automated tests on the instrumented executable.
- The build infrastructure that RE owns and allowed QA to run code coverage tests on it.
Almost always, the build infrastructure can be ruled out as the cause of failure. If that were the case, the entire build setup would catch FIRE. So, if the results are not there, please call QA points of contact who are Murali and Tim who know the reasons for failure.
Many times, the reason for failure happens to be the way we are picking up bleeding edge nightly trunk code base to create the executable. Creating an instrumented executable adds a lot more sensitivity to the product compared to an optimized build. One way, it is good that we are able to find some bugs that could not have been found with regular optimized builds.
In order to improve the infrastructure resiliency, we are making continuous improvements to the way we run tests. Recently, we created serialized Mochitest target which runs the big 72K+ tests in smaller chunks serially so that even if there is a crash/hang of Firefox during test run, it is localized only to the smaller chunk.
In few more days, if you are building and running tests on a regular nightly executable [ as a community volunteer ] you can use 'make mochitest-serial' to run Mochitests in smaller batches instead of your current option 'make mochitest-plain'.
Coverage reports are provided regularly at the following locations.
- http://people.mozilla.com/~mnandigama/codecoverage_html/index.html
- http://people.mozilla.com/~mnandigama/jscoverage-report/jscoverage.html
Thank you.
Comments
The JS report is empty when I view it. Is it supposed to have data?