What questions do we want answered.

Practices for Deciding what to Test and what to Automate

We test in order to get certain questions answered. We automate tests when we want to get those questions answered with a minimum of human thought. The idea is that we have the human figure out what questions to ask and how to answer them and let the machine do the grunt work.

Choosing what tests to automate is always the result of a cost-benefit analysis. A.k.a. weighing the benefits against the costs. In most projects you can't test and automate everything, so you want tests that answer the questions you want answered the most. We want to automate the tests that will give us the biggest bang for the buck.

I typically document the tests by the questions the tests are designed to answer. A set of tests may be designed to answer whether or not the current build is worth subjecting to further tests. i.e is the current build minimally acceptable. I have some sample questions below.

As you look at these questions below in each case you want to ask yourself to other questions "Do we care?" and  "How much do we care?". Some of these questions are not important to us because of the specific project that we are working on. In almost all cases there is a priority to what questions you want answered first.

Typical Questions

• Do various classes and functions in the code work correctly in isolation. Typically tests that are designed to answer this question fall under the label of unit tests.
• How does a given protocol handle bad state changes. Typically tests that are designed to answer this question referred to as Short Tour Tests.
• Does the SUT (System Under Test) deploy correctly. Typically tests that are designed to answer this fall under the label of smoke tests.
• Does the same code produce the same result when invoked from different platforms.
• Does the current build meet some minimal criteria for being deployed and tested further? Typically tests that are designed to answer this question fall under the label of smoke tests.
• How does the (SUT) perform under specific loads. Tests that answer those questions are typically referred to as performance tests
• Does the SUT past the same test it used to pass? Tests that are designed to answer this question are typically referred to as regression tests.

Once you have decided what questions are important and how important they are, you need to design an automated  testing regimen that answers those questions in that order of priority.

I want to emphasize that the questions to be answered and their priorities are key factors in determining the effectiveness and efficiency of the testing process. If you are creating unit tests, integration tests, and performance tests and they don't answer the questions you need to have answered, you are now maintaining unit tests, integration tests, and performance tests that do not contribute to your business. In other words, you are generating overhead and possibly technical debt.

Once you know the questions you want answered and how important they are to you, it's time to figure out what testing types you will use get those answers and what actions your tests will perform.

One key implication of this is that when you're creating individual tests try to keep them to answering one question at a time. As the system evolves answering that one question may become more complicated or involved but evaluating whether or not the test is well designed and executed can always be evaluated by determining whether or not it allows you to answer that one question.

General rules to keep in mind

Test Early/Test Often

The earlier in the development/testing lifecycle that you get the answer to the question, the less expensive it is to respond to.

Don't buck the tide

Using tools in the way they are designed to be used is far more effective than cramming the tool to fit.

For example: Maven is an automated build tool that is designed around some methodologies that they consider best practices. If you are not taking the actions that Maven expects you to take you can end up spending a great deal of time working on or against Maven rather than working on your tests. By the same token, if you do adhere to what Maven expects, things will be fairly straightforward.

Don't improve the process until you know what you are improving

You can't improve the process unless you have numbers!

You are adding test an existing process it makes sense to test the areas that have the most bugs are the least stable. If you don't have coverage analysis of the code you can't say how much of the code is actually exercised by the testing process. If you don't know what the current performance level of the product is you have no starting point for figuring out where to speed it up.

By the same token, metrics are at best a guideline, if you use a metric for what it's not designed for, it can be dangerous to your sanity.

Context is Decisive

“A best practice is always a best practice; Except when it's not.”

I have recently been involved in a contract that has forced me to really really look at testing practices and how I explain testing practices. 

This particular department of a very large company gives lip service and action to a lot of "Best Practices" in the world of development and testing. SCRUMs, TDD, BDD, etc. are all the words you see floating around in the air there. They are saying all the right things, and even taking many good actions, and the end result is almost worse than if they took no actions at all.

It is, in my world the best illustration of a very basic principle, context is decisive. Another way of saying that is that a best practice is always a best practice, except when it's not.

To illustrate the point: 

In the context of body part the above is a Finger

In the context of number the above is One

Same picture, different interpretation.

This is particularly notable when you start talking about using "Best Practices" out of context.

When I was starting out as a software engineer I worked for an Foliage Software Systems. They had a significantly higher than average success rate with projects and a higher than average satisfaction level of their customers. One of the things that Foliage did that made a difference was that they had a playbook of processes. So they would have a preliminary engagement with the customer where the lead on the project would discover what kind of existing coding and communication processes they already had in place and use the playbook to select the Foliage development and communication processes to fit with what the customer was doing. This came about because they discovered early on that merely producing a technically successful project was not enough to produce a satisfied client. If the development process was not understandable by or compatible with the client they could have an unsatisfied client. The communication process was not understandable by or compatible with the client, they could have an unsatisfied client. And they worked on FAA and FDA certified projects as well as many, less stringent, markets.

By choosing the process to match the context they ended up with a higher number of successful projects.

As far as I can tell the majority of the TDD is Dead discussion comes down to, if you use the TDD methodology out of context, it gets messy and doesn't deliver on what you need. I completely agree. 

I use TDD a lot. It allows me to shake out the design of the individual modules at a low level and alerts me to the impact of re-factoring. I don't test getters and setters. I don't test methods that just delegate to another method. I don't write code for functionality that isn't tested. And I don't forget that the unit test provides me with a minimal contract regarding the usability of the module. For me the context of TDD is to answer some key questions such as "Does this module meet some minimum robustness requirement?", "Do I know how I'm going to use this module?" , "Did I break anything when I re-factored the classes?". So I use TDD to drive the design and implementation of the individual modules and leave me with a test that documents some ways to use the module, specifies the core functionality of the module, and acts as a tripwire for detecting the impact of re-factoring. In that context, TDD is quite successful.

When I was doing research I came across a website describing the school of thought called Context Driven Testing. You may or may not agree with all of the discussions on the webpage. But the heart of it in the basic principles is one of the clearest statements around best practices that I've ever see.

The Basic Principles of Context-Driven Testing

1. The value of any practice depends on its context.
2. There are good practices in context, but there are no best practices.
3. People, working together, are the most important part of any project’s context.
4. Projects unfold over time in ways that are often not predictable.
5. The product is a solution. If the problem isn’t solved, the product doesn’t work.
6. Good software testing is a challenging intellectual process.
7. Only through judgment and skill, exercised cooperatively throughout the entire project, are we able to do the right things at the right times to effectively test our products

In the next entry I will discuss  what questions to ask to determine what to test, and what to automate.