Truth is a model (from Neil Gershenfeld)

The most common misunderstanding about software testing is that testers seek and find truth (corollary – testers find faults). They don’t — they make and test models.

Kepler packing Platonic solids to explain the observed motion of planets made pretty good predictions, which were improved by his laws of planetary motion, which were improved by Newton’s laws of motion, which were improved by Einstein’s general relativity. Kepler didn’t become wrong because of Newton being right, just as Newton didn’t then become wrong because of Einstein being right; this succession of models differed in their assumptions, accuracy, and applicability, not their truth.

This is entirely unlike the polarizing battles that define so many areas of life: either my political party, or religion, or lifestyle is right, or yours is, and I believe in mine. The only thing that’s shared is the certainty of infallibility.

Building models is very different from proclaiming truths. It’s a never-ending process of discovery and refinement, not a war to win or destination to reach. Uncertainty is intrinsic to the process of finding out what you don’t know, not a weakness to avoid. Bugs are features — violations of expectations are opportunities to refine them. And decisions are made by evaluating what works better, not by invoking received wisdom.

These are familiar aspects of the work of any scientist, or baby: it’s not possible to learn to talk or walk without babbling or toddling to experiment with language and balance. Babies who keep babbling turn into scientists who formulate and test theories for a living. But it doesn’t require professional training to make mental models — we’re born with those skills. What’s needed is not displacing them with the certainty of absolute truths that inhibit the exploration of ideas. Making sense of anything means making models that can predict outcomes and accommodate observations. Truth is a model.

This post is a copy of Neil Gershenfeld‘s response to the 2011 Edge question.  I replaced the words ‘science’ and ‘scientists’ with ‘software testing’ and ‘testers’ in the first sentence.  I also added the phrase in parenthesis in the first sentence.  You could replace the title and the last sentence with ‘Expected results is a model’.  This should explain why testers don’t/shouldn’t try to determine (or even worse prove) whether the software works.  It also describes that ‘bugs’ are not an indication of (human/developer) failure ( When working with good developers, very few defects are of that category).

The agile movement’s principle of making developers responsible for testing is a good idea.  While a developer can be confident of his technical skills (in programming/computer science), meeting a user’s needs, working with complex technology and large systems requires an empirical approach as followed by scientists.

I wrote an article for Tea Time with Testers with concepts similar to this post – a collection of links from the edge’s 2011 question on how to think about software testing.  If the link doesn’t work, you can go to the archives of Tea Time with Testers (!magazines/galleryPage) and look for the July 2013 issue.

‘Lean Startup’ definition of value

In the book “The Lean Startup Method”, Eric Ries, as CTO/co-founder of IMVU, describes an early experience with the startup.  After months of product development they realized that their customers weren’t interested in their product.  He realized that while he could console himself with the benefit of learning from the failure, such learning was little consolation to employees and investors.

The key concept for startups is what Ries defines as “validated learning”.

How do you validate learning?
You create increments of your software which you release to customers.  You track the response of specific groups of customers.  For example, you should be able to generate data which shows that after introducing the feature ‘simple login’ in February, the adoption of customers who used that feature has greatly increased compared to customers who don’t have that feature (as opposed to showing a general graph of customer adoption.)

Ries also introduces the concept of the scientific method and ‘experiments’.  This is how he describes an ‘experiment’.

“It begins with a clear hypothesis that makes predictions about what is supposed to   happen.  It then tests those predictions empirically. “

He gives the example of Zappos, the world’s largest online shoe store:

When starting the store, the founder, took pictures of shoes from stores and posted them online.  The question he was trying to answer was: “Is there already sufficient demand for a superior online shopping experience for shoes?”  The important takeaway from the example is that you should build a “minimalistic” product to conduct your experiment (as opposed to early e-commerce startups which invested in warehouses, inventory and a final product).

Definition of value

In the Lean Startup, anything which doesn’t result in validated learning is waste.  Instead of creating product features, you can create a ‘facade’ (see MVP below).  This is then used to get feedback from customers.  The feedback is in the form of actual customer adoption, not as a survey or a focus group.

While soliciting customer feedback is an important part of agile, sometimes teams may be a bit forgiving about the feedback they get.  Internal stakeholders may be used as proxies for customers (in agile).  The Lean Startup is unambiguous in the feedback (validated learning).  Teams must be able to check each user story with ‘validated learning’.  Without that the story is incomplete.

This is enough to understand the concept of ‘value’ in the Lean Startup.  I will write more about the ‘Lean Startup’ later.  There are a few additional concepts which are relevant to this post which are explained in the following paragraphs.

Additional concepts

Minimum Viable Product (MVP) : A Minimum Viable Product (MVP) is the smallest needed to sell to potential customers.  A good example of this is the start of Groupon.  Groupon started by rapidly assembling a set of deals without a lot of focus on the presentation.  The idea is to get rapid feedback on the concept which can be refined based on customer feedback. (The definition of MVP used in the book is from Frank Robinson of PMDI ).

Startup: What is a startup?  The book cover includes a definition of a startup, “an organization dedicated to creating something new under conditions of extreme uncertainty.  This is just as true for one person in a garage as it is in a group of seasoned professionals in a Fortune 500 boardroom.  What they all have in common is a mission to penetrate the fog of uncertainty to discover a successful path to a sustainable business.”  Although this may be partly marketing, the definition includes any type of software project/product.  In the book Ries gives an example of Intuit using his concepts.  He gives many other example of large companies.

Agile vs. Lean Startup: It might seem that the Lean Startup doesn’t really add anything new to agile.  One major difference is that the Lean Startup is insistent on ‘validated learning’.  In the case of agile, customer feedback is critical.  However, in smaller organizations or for products which are not web based, there may be limited customer feedback.  The product owner and the team can also be used as proxies for the customer.  Not so in the Lean Startup.  The other important difference is Lean Startup’s focus on business returns/value.  A key question that Lean Startup teams need to answer is to show growth in existing customers or growth in adoption by new customers – without smoke and mirrors presentations.  Similarly the Lean Startup also encourages teams to validate their business model by creating MVPs which are focused on the business and business processes, not necessarily the technology.

Understanding lean manufacturing ‘value’

Every other day, in the context of software testing, there is a discussion about ‘value’.  “Do testers provide value”, “Testers should provide value” and the related, “Can we eliminate testing?” or “Testing is dead”.  In many cases, the word “value” is used colloquially (gold is valuable; dirt isn’t) and even casually (value is value), without much thought to what it really means.  In the context of lean (manufacturing), value has a very specific definition/meaning.  I think it is essential to understand the meaning of ‘value’ in lean, before using the word in other [technical] contexts.  In this post I will not refer to software development.  For this discussion you can think of creating a product like a bolt in a factory or parts of a car.  I will also refer to the process of visiting a doctor.

In the context of lean, any activity used to produce a product or a service can be one of three categories:
1. value added
2. non value added
3. non value added but necessary

There are some activities which are obviously non-value added.  In the context of a visit to the doctor, a patient waiting for the doctor is non-value add.  Mistakes in a doctor’s lab report, which requires rework, is non-value add.  In the context of manufacturing, defective parts requiring rework is non-value add.

The categories for some activities are not obvious, e.g., suppose an administrator delivers lab reports to the doctors in the clinic – is that value add?  Can you prevent delivering the reports?  Suppose you keep records for defective parts in a manufacturing facility – is that value add?  Can you avoid keeping those?

In lean manufacturing, in order to qualify as value-add, an activity must meet the following three criteria:
1. It must change form or function of the product or service
2. The customer must be willing to pay for it
3. It must be done right the first time

Even with this definition, it can be challenging to figure out if an activity is value-added.

In the context of lean, when we discuss if the customer is willing to pay for it, we are not discussing personal preference.  It doesn’t matter if the customer likes red pens or not.  In the case of manufacturing, whether a customer ‘likes’ an item is a problem of it’s design or marketing.  That is not relevant to the discussion of ‘value’.  In the case of software, engineers are responsible for the design, and it is important that target customers see ‘value’ in the design.  I won’t discuss software in this post.  It’s enough to keep in mind that a customer’s willingness to pay (in lean manufacturing) is more related to wasteful activities, such as defects or excess inventory.

There are some activities which are not value-added, e.g., they do not change form or function of products, but which are necessary.  For example, sales and marketing of products or services.  Although these will not satisfy the three conditions for value-add, a business will not survive without these activities.

Why is value important (in lean manufacturing)?
When people pursue process improvement or try to make processes more efficient, they focus on the small percentage of value added activities – “make it faster”.  Instead, eliminating NVA can give much higher gains with less effort. Typically only 10 to 15 percent of steps in a process add value, and more often than not, these steps represent as little as 1 percent of the total process time (From Lean Six Sigma for Dummies).  Consider a typical visit to the doctor’s office, there are large periods of waiting with relatively little productive time spent on filling forms and talking to the doctor.  Focusing on reducing the wait times will result in higher gains compared to improving the time taken by the doctor.

Here are some of the ways to eliminate waste in a doctor’s office.
– have broad roles for admin staff, eliminating unnecessary hand-offs
– making sure physician’s have all the information needed to focus on the patient
– avoiding rework by different admin staff, the nurse and physician
– using technology to avoid mistakes and rework

Identifying activities which create value is the first step in applying lean (More on lean later).

The concepts of lean can be applied to any industry.  If you would like to know more, you may want to start with understanding how it is applied to health care/hospitals/doctors.  If you don’t have experience with manufacturing it may be easier to understand how lean concepts are applied to a doctor’s office.  There are many references on the subject available on the Internet.

Leadership in a large test organization

The main keynote in Stareast 2012 was by Keith Klain of Barclays Capital Global Test  Center.  Zeger Van Hese has written a nice account of Keith’s talk.

Keith’s talk was an inspring story of personal leadership and leadership shown by an organization, Barclays Capital.  There were important lessons for test managers and leaders:

  1. Attract and retain the best talent. In testing it is often a challenge to attract and retain the best.  In Barclays, Keith made sure that testing was considered the ‘place to be’.  This probably worked better in the case of Barclays since it is a large organization and people can move between departments.  It might have also helped that IT overall (test and dev) is probably a strong function in Barclays.  If you have a large test organization (or a small one) this is a key *value* and challenge – making sure that you can attract and retain the best.
  2. Keith mentioned that he has a very flat organization.  Test managers spend 80% of their time testing.
  3. The flat organization ensures that testers don’t obsess over the next better title or grade.
  4. The organization has a big focus on training.  This isn’t just a “HR initiative”.  It was clear that Keith is deeply involved with this personally and not something he “manages”.  Barclays provides various training including in tools such as QTP.  Keith has arranged for extensive training by James Bach (Satisfice).  Given the tight budgets in the last few years, managers always start groaning when anyone mentions training.  In Barclays team members took the initiative to provide training to others.

In addition to the management lessons, there were many personal lessons:

  • One of the key lessons was that testers should be responsible for how they are perceived by the organization.  Too often, testers don’t spend time reading or learning about testing.  Given that there are very few formal degrees in testing, this seems like a sure way to ignominy.
  • In the last few decades one of the phrases which has become common place is ‘direct but sensitive’.  In Keith’s talk it seemed much more  genuine when he said that sometimes you have to ‘speak your mind, that’s OK, as long as you do it respectfully.’
  • Another core value which appealed to me was that of ‘questioning everything’.  Don’t just do your job.  Make sure you question everything.’  The lesson in leadership and management is that it helps when this message comes from the head of an organization.

This talk was not about idealogy or management jargon.  It was about values and setting the tone for an organization.  It was about hard work, responsibility and accountability.  It was about taking pride in your work and excellence.

What is your quality philosophy?

Sometime back, on the software testing group on linkedin, there was a discussion/question:
What is your Quality philosophy?

Here is my response:

  • Quality = customers want to use our products + low support issues.
  • While customer feedback is critical, we independently take efforts to judge the probability that the product/project has important undiscovered problems.
  • Individuals have character. They take responsibility for the quality of their work. They
    use feedback and introspection to continuously improve.
  • Skills in development, testing, documentation (engineering) are critical and paramount.
  • We develop in small increments
  • We actively engage with customers throughout the development cycle and allocate budget to this activity to ensure success.
  • Managers (first line) have a technical background and can be hands-on if necessary. Managers have character. They use personal observation as the basis for managing teams.
  • We have the best technical support team in the world. We have a system for using support information to provide feedback to the engineering teams.
  • When measuring project or people performance, numbers are not a substitute for judgement and observation. Numbers may indicate a need for further investigation.


Here is how I would summarize what I said in the previous paragraphs:
Quality philosophy – keep striving for improvement in knowledge and skills.  All actions must demonstrate character.