Latest Developments in Code Coverage Exercise
I have been working on the fully automated code coverage infrastructure for both C/C++ and the JavaScript code portions on Firefox trunk. This process is the same for branches as well.
Few months ago, catlee and joduinn from release engineering have generously helped to integrate the C/C++ code coverage setup in to buildbot infrastructure.
However, our original plan to generate a regular weekly code coverage run that provides consistent results ran into some teething troubles. The reason for this trouble has more to do with us picking up a bleeding edge trunk code base and instrumenting it and running massive test suites like 'Mochitest' harness on it. As you are probably aware, an instrumented build is more sensitive to problems that can be forgiven by an optimized build. If Mochitest harness crashes during the test run, depending on where it crashed [ the browser and causing it to quit ], the remaining part of tests end up not being run at all.
So, a new test target called mochitest-serial is created [ https://bugzilla.mozilla.org/show_bug.cgi?id=502689 ] that would split the humongous Mochitest suite into smaller parts and run them serially. This way, even if a portion of the suite crashes/hangs, the subsequent chunks can continue running the remaining tests. This should get checked-in very soon in the current or modified form.
Once this is integrated into R.E. framework, the weekly runs on the buildbot should yield code coverage results that have very less fluctuation. In the meantime, I would be generating code coverage results and posting them in the usual place.
However, the main point of this post is to inform you all that we have successfully completed the fully automated instrumentation -> results_collection portion for JS portions now. Catlee is integrating this into buildbot framework as we speak. So, the results of javascript code coverage would also be provided on a regular basis along with C/C++ code coverage results.
Please note that all javascript files in the Firefox product can not be instrumented. The javascript files in 'greprefs' & 'defaults' directories should not be instrumented and so coverage data on the javascript files in those directories can not be provided. Please find the first posting of javascript code coverage results.
Please be patient - the javascript code coverage results take time to load since it needs to parse a massive JSON file.
From this week onwards, every Tuesday , you should be able to check the links given below to see the latest C/C++ and Javascript code coverage results.
Ultimately, catlee's efforts to run all tests in parallel on multiple machines would reduce the cycle time significantly. After that task is accomplished, we can update the code coverage results more than once a week.
Few months ago, catlee and joduinn from release engineering have generously helped to integrate the C/C++ code coverage setup in to buildbot infrastructure.
However, our original plan to generate a regular weekly code coverage run that provides consistent results ran into some teething troubles. The reason for this trouble has more to do with us picking up a bleeding edge trunk code base and instrumenting it and running massive test suites like 'Mochitest' harness on it. As you are probably aware, an instrumented build is more sensitive to problems that can be forgiven by an optimized build. If Mochitest harness crashes during the test run, depending on where it crashed [ the browser and causing it to quit ], the remaining part of tests end up not being run at all.
So, a new test target called mochitest-serial is created [ https://bugzilla.mozilla.org/show_bug.cgi?id=502689 ] that would split the humongous Mochitest suite into smaller parts and run them serially. This way, even if a portion of the suite crashes/hangs, the subsequent chunks can continue running the remaining tests. This should get checked-in very soon in the current or modified form.
Once this is integrated into R.E. framework, the weekly runs on the buildbot should yield code coverage results that have very less fluctuation. In the meantime, I would be generating code coverage results and posting them in the usual place.
However, the main point of this post is to inform you all that we have successfully completed the fully automated instrumentation -> results_collection portion for JS portions now. Catlee is integrating this into buildbot framework as we speak. So, the results of javascript code coverage would also be provided on a regular basis along with C/C++ code coverage results.
Please note that all javascript files in the Firefox product can not be instrumented. The javascript files in 'greprefs' & 'defaults' directories should not be instrumented and so coverage data on the javascript files in those directories can not be provided. Please find the first posting of javascript code coverage results.
Please be patient - the javascript code coverage results take time to load since it needs to parse a massive JSON file.
From this week onwards, every Tuesday , you should be able to check the links given below to see the latest C/C++ and Javascript code coverage results.
- C/C++ : http://people.mozilla.com/~mnandigama/codecoverage_html/index.html
- JavaScript : http://people.mozilla.com/~mnandigama/jscoverage-report/jscoverage.html
Ultimately, catlee's efforts to run all tests in parallel on multiple machines would reduce the cycle time significantly. After that task is accomplished, we can update the code coverage results more than once a week.
Comments
We see it with Valgrind relatively often -- bugs that occur natively but not under Valgrind, or vice versa.