Different Strokes
Standard Disclaimer : What is written here are author's personal opinions and are not endorsed by any organization to whom the author provides services.
Now that I am done with the standard disclaimer, let me share with you some of my thoughts after reading the following feature @ http://research.microsoft.com/en-us/news/features/nagappan-100609.aspx
I agree with many aspects of the Featured engineer's findings but I have qualms with the way they are presented.
Let me repeat the age old knowledge, 'conventional software-engineering wisdom' is as nebulous a phrase as 'universal harmony'. There is no accepted dogma in this area. There are 'Microsoft' software-engineering practices,'Google' software-engineering practices, 'Oracle' software-engineering practices and so on ... Just because we read some graduate course text books that talk about software-engineering practices, it does not mean that they are definitive. The article's author claims that Microsoft ESM group studies the conventional software-engineering wisdom. This is a non-pragmatic statement considering that the group consists of a lot of researchers who know that there is no 'conventional-way'.
In this context, let me share a pertinent story...which is true.
One of my friends did his Ph.D. in physics from Princeton University and during the first year, he saw a brown Squirrel and made a statement 'Squirrels are brown!'. Next year his statement was more contained 'Squirrels on the Princeton campus are brown'. The following year his statement was more measured 'Some Squirrels on Princeton campus are brown'. In the final year of his graduate studies his comment was 'I saw a brown Squirrel on Princeton campus'.
I hope the statement 'Code coverage measures how comprehensively a piece of code has been tested' is article writer's paraphrase [ and not of the featured Engineer ]. In this part of the world, in some companies I worked, like Sun, Oracle and Mozilla, code coverage is not used as a measure of comprehensiveness, not at least in the groups I worked in. Here, every one knows that having 100% code coverage does not mean that you have found all your bugs. But having 20% code coverage means that some thing is seriously amiss! Code coverage does not give good news. It indicates potential gaps that we are not at all testing. If you are trying to disprove the assumption that 'Higher code coverage means lesser post-release failures in the field' , then I humbly advise that it is time to get your tuition fee back [ if only your college offers satisfaction guarantee ] and also please visit https://wiki.mozilla.org/Firefly . I have seen more people who believe in 'Spaghetti Monster' than those who believe in the bug finding-ness of code coverage.
I would have been seriously shocked too ... if Microsoft developers danced to the tune of 'Kumbaya' after learning this triviality.
The other issue that rattled my brain is the emphasis on Test Driven Development [TDD]. If you were a nascent start-up writing your first shopping cart application checking your code into source code control for the first time, then 'yes' TDD is the new cool kid on the block. But is this the right approach for Microsoft!!
I heard a joke from an industry veteran recently 'Why Microsoft takes years to release a product when the God completed the whole creation in just seven days ? ... Because God does not have to worry about backward compatibility'
What would it take to retrofit the decades worth of legacy code into the TDD model !!
The right approach is in fact 'Change Centric Test Development'.
What is change centric test development ?
Incidentally, Nagappan practically explains the Change Centric testing infrastructure in the 'Proving the Utility of Assertions' section sans linking it to code coverage data.
Rest of the article deals with Organization Design and its impact on the development process.
While the beginning of the article starts with a sense of universality, the article writer admits [ at the end ] that the findings are derived from studying MS groups and that Nagappan would like to correlate it to other environments. At the same time, the article also concludes that they have solved the 'software-engineering myth buster' .... perhaps an internal Microsoft one!!
Now that I am done with the standard disclaimer, let me share with you some of my thoughts after reading the following feature @ http://research.microsoft.com/en-us/news/features/nagappan-100609.aspx
I agree with many aspects of the Featured engineer's findings but I have qualms with the way they are presented.
Let me repeat the age old knowledge, 'conventional software-engineering wisdom' is as nebulous a phrase as 'universal harmony'. There is no accepted dogma in this area. There are 'Microsoft' software-engineering practices,'Google' software-engineering practices, 'Oracle' software-engineering practices and so on ... Just because we read some graduate course text books that talk about software-engineering practices, it does not mean that they are definitive. The article's author claims that Microsoft ESM group studies the conventional software-engineering wisdom. This is a non-pragmatic statement considering that the group consists of a lot of researchers who know that there is no 'conventional-way'.
In this context, let me share a pertinent story...which is true.
One of my friends did his Ph.D. in physics from Princeton University and during the first year, he saw a brown Squirrel and made a statement 'Squirrels are brown!'. Next year his statement was more contained 'Squirrels on the Princeton campus are brown'. The following year his statement was more measured 'Some Squirrels on Princeton campus are brown'. In the final year of his graduate studies his comment was 'I saw a brown Squirrel on Princeton campus'.
I hope the statement 'Code coverage measures how comprehensively a piece of code has been tested' is article writer's paraphrase [ and not of the featured Engineer ]. In this part of the world, in some companies I worked, like Sun, Oracle and Mozilla, code coverage is not used as a measure of comprehensiveness, not at least in the groups I worked in. Here, every one knows that having 100% code coverage does not mean that you have found all your bugs. But having 20% code coverage means that some thing is seriously amiss! Code coverage does not give good news. It indicates potential gaps that we are not at all testing. If you are trying to disprove the assumption that 'Higher code coverage means lesser post-release failures in the field' , then I humbly advise that it is time to get your tuition fee back [ if only your college offers satisfaction guarantee ] and also please visit https://wiki.mozilla.org/Firefly . I have seen more people who believe in 'Spaghetti Monster' than those who believe in the bug finding-ness of code coverage.
I would have been seriously shocked too ... if Microsoft developers danced to the tune of 'Kumbaya' after learning this triviality.
The other issue that rattled my brain is the emphasis on Test Driven Development [TDD]. If you were a nascent start-up writing your first shopping cart application checking your code into source code control for the first time, then 'yes' TDD is the new cool kid on the block. But is this the right approach for Microsoft!!
I heard a joke from an industry veteran recently 'Why Microsoft takes years to release a product when the God completed the whole creation in just seven days ? ... Because God does not have to worry about backward compatibility'
What would it take to retrofit the decades worth of legacy code into the TDD model !!
The right approach is in fact 'Change Centric Test Development'.
What is change centric test development ?
- Identify the extent of code coverage provided by existing test suites
- Identify the nature of bugs filed/fixed on the code base and weigh it against the coverage effectiveness. If you have 100% code coverage on a module but over the last year 80% of bugs are found by customers, then your test suite is rotten. Throw it our, rinse and restart.
- Create test-case to source-file/functions/methods mapping .. create test-case to bug mapping ... create bug to source file mapping. Any reasonable source code control system and bug management system and a data base to store mappings would be sufficient.
- Track daily code changes and check if you have corresponding tests in the test suite from above steps. If not , create a test case and DO NOT check in your code till you have one.
Incidentally, Nagappan practically explains the Change Centric testing infrastructure in the 'Proving the Utility of Assertions' section sans linking it to code coverage data.
Rest of the article deals with Organization Design and its impact on the development process.
While the beginning of the article starts with a sense of universality, the article writer admits [ at the end ] that the findings are derived from studying MS groups and that Nagappan would like to correlate it to other environments. At the same time, the article also concludes that they have solved the 'software-engineering myth buster' .... perhaps an internal Microsoft one!!
Comments