My 2010 keynote at the Google Test Automation conference considered the dimensions of testability and its implications. Click here for the slides. Click
here to view the video.
This presentation is serialized in following posts.
- Part 1: Testability: What is it?
- Part 2: Controllability and Observability
- Part 3: Accidental Untestability
- Part 4: White Box Testability
- Part 5: Black Box testability (not posted yet)
- Part 6: The Role of Test Automation (not posted yet)
- Part 7: Why Testers get No Respect (not posted yet)
The incredible advances in computing capacity we’ve become blasé about (even impatient and demanding) would not have been possible without standard test features in all kinds of digital logic devices. “Design for Testability” is critical for all kinds of digital devices but little known outside of hardware design and manufacturing. The nearby graphic shows this nicely (I would like to provide appropriate attribution, but couldn’t determine the author.)
Could a similar approach help to make software cheaper, better, faster? I thought so, but it turned out that like many promising hardware/software analogies, the software problem was unbounded and more complex.
To begin with, what is “software testability” and why does it matter?
Testability is the degree of difficulty of testing a system. This is determined by both aspects of the system under test and its development approach.
- Higher testability: more better tests, same cost.
- Lower testability: fewer weaker tests, same cost.
Let’s assume the following about software development. Other things being equal, and on average:
- Development (including testing) occurs with a fixed budget, so the key question is how to optimize the value produced.
- Testing adds value by minimizing the bugs in a released system.
- Sooner is better: We’re better off when we release our software product sooner.
- Escapes are bad: The old a bug gets, the nastier (more expensive) it becomes.
- Fewer tests means more escapes: Suppose our tests have 1:100 odds of finding a bug and there are 1,000 latent bugs in our system. We need to run at least 100,000 tests to find these bugs. But suppose we run only 50,000 tests and release; we’ll probably ship with about 500 latent bugs.
Testability determines the limit to which the risk of costly or dangerous bugs can be reduced to an acceptable level.
That is, poor testability means you’ll probably ship/release a system with more nasty bugs than is prudent.
About 130 individual factors that contribute to testability. The effect of all this can be measured with two ratios:
- Efficiency: average tests per unit of effort. Or, much testing can we get done with the time, technology, and people on hand?
- Effectiveness: average probability of killing a bug per unit of effort.
Improved testability means we can do more testing and/or increases the odds we’ll find a bug when we look.