Truth is a model (from Neil Gershenfeld)

The most common misunderstanding about software testing is that testers seek and find truth (corollary – testers find faults). They don’t — they make and test models.

Kepler packing Platonic solids to explain the observed motion of planets made pretty good predictions, which were improved by his laws of planetary motion, which were improved by Newton’s laws of motion, which were improved by Einstein’s general relativity. Kepler didn’t become wrong because of Newton being right, just as Newton didn’t then become wrong because of Einstein being right; this succession of models differed in their assumptions, accuracy, and applicability, not their truth.

This is entirely unlike the polarizing battles that define so many areas of life: either my political party, or religion, or lifestyle is right, or yours is, and I believe in mine. The only thing that’s shared is the certainty of infallibility.

Building models is very different from proclaiming truths. It’s a never-ending process of discovery and refinement, not a war to win or destination to reach. Uncertainty is intrinsic to the process of finding out what you don’t know, not a weakness to avoid. Bugs are features — violations of expectations are opportunities to refine them. And decisions are made by evaluating what works better, not by invoking received wisdom.

These are familiar aspects of the work of any scientist, or baby: it’s not possible to learn to talk or walk without babbling or toddling to experiment with language and balance. Babies who keep babbling turn into scientists who formulate and test theories for a living. But it doesn’t require professional training to make mental models — we’re born with those skills. What’s needed is not displacing them with the certainty of absolute truths that inhibit the exploration of ideas. Making sense of anything means making models that can predict outcomes and accommodate observations. Truth is a model.

This post is a copy of Neil Gershenfeld‘s response to the 2011 Edge question.  I replaced the words ‘science’ and ‘scientists’ with ‘software testing’ and ‘testers’ in the first sentence.  I also added the phrase in parenthesis in the first sentence.  You could replace the title and the last sentence with ‘Expected results is a model’.  This should explain why testers don’t/shouldn’t try to determine (or even worse prove) whether the software works.  It also describes that ‘bugs’ are not an indication of (human/developer) failure ( When working with good developers, very few defects are of that category).

The agile movement’s principle of making developers responsible for testing is a good idea.  While a developer can be confident of his technical skills (in programming/computer science), meeting a user’s needs, working with complex technology and large systems requires an empirical approach as followed by scientists.

I wrote an article for Tea Time with Testers with concepts similar to this post – a collection of links from the edge’s 2011 question on how to think about software testing.  If the link doesn’t work, you can go to the archives of Tea Time with Testers (!magazines/galleryPage) and look for the July 2013 issue.

Step closer to exploratory testing

Many testers cannot imagine testing without using testcases.  In the last 10 to 15 years testing has been clearly explained by a group of testers representing themselves as the context driven testing school.  Many of them are also associated with the association of software testing.

When traditional testers are introduced to the concepts of the CDT, especially the idea of exploratory testing, it can take a long time to cross the chasm between traditional (writing testcases, getting them approved and then executing them) and exploratory testing.

I understood exploratory testing quite late.  Before I knew what was exploratory testing I worked with some good testers.

Although we didn’t know what was exploratory testing, I think we intuitively followed some of the principles.  I have described what we did so that it can be used as a stepping stone by testers who want to move towards exploratory testing.  This is a description of how the testers I worked with (in a previous job) tested.

How we tested

We hired testers who were domain experts (the domain required strong technical knowledge).  Some testers had used competitive products in their previous jobs.  Developers were strong programmers, some had knowledge in the domain.
We had very strong product designers who created great functional specifications.

QA got involved as early as they could.  If there was even the slightest scent of a discussion (much before the actual release), QA would get involved.  A lot of time was spent understanding the specs and asking for changes.

Developers would start writing code as soon as they could.  They would create detailed design docs/diagrams.  The feature team would discuss these designs.

QA created test plans in excel.  Testcases were one liners.  These were reviewed by a team and corrections were made to the doc.   It was difficult to review each others testcases since each feature was quite complex.  However, there would be some good suggestions.

The minute a build was available, QA would start using the build.  They didn’t have to refer to the testcases.  They could find defects just using the software.  When they needed ideas they would refer to the testcases.

After an initial frenzy of logging defects (say a month), developers started fixing defects.  QA would continue to verify fixes and re-testing functionality.

We would have bug bashes with the entire team testing each feature.

Testers would review all defect fixes for their feature.  Their testing was based on what was fixed.

The senior manager would sometimes use his judgement about the quality of testing of a particular feature.  In some cases we would assign another tester to ‘help find more defects’.

The managers would view defect trends.  However, not many people read too much into it.

A month or two before the release, a few experienced team members formed a committee which would control what fixes went into the product.

For the good testers most of the defects found late in the cycle were due to regression (as a result of defect fixes).  If a defect was found by someone other than the feature tester, good testers knew whether it was something they had tested or not.  In some cases they had to refer to their notes to trace when they had tested something.

Final Notes

Testcases – really didn’t mean much
Domain knowledge – this made a huge impact in this team.  Note that in many software there may not be strong domain knowledge, e.g., simple banking application, an instant messenger product.
Defect trends – nothing profound about the trends.  I can summarize as: log many defects, at some random point you find less than before, during the end game force the graph down (by not allowing fixes)
Release criteria – we made sure all important defects were fixed.  What this means is that criteria didn’t really matter.  Important defects were discussed and triaged.  What made this work is that we controlled what was fixed a month or two prior to release.

The most important factor in testing was the knowledge and skill of the tester.  Testers spent more than 95% of the time working with the software or ‘studying’ information such as developer documents and defect fixes.

What we didn’t do and could have bitten us

Looking back, if we understood exploratory testing, we would/should have done things differently.

  • We focused mainly on functionality.  We did spend some time on performance.  However, we should have thought about other non-functional tests.
  • We should have spent much more time asking what-if questions.  Some of this was done implicitly.
  • We should have designed more complex tests
  • We should have spent more time discussing tests (semi-formally)
  • We should have done more brainstorming
  • We should have used more formal techniques such as scenarios
  • We should have spent more time working with the developers to understand what was behind the functionality

Despite what we didn’t do, what worked for us is – being able to hire super smart testers and allowing them to do their best.