Prevent Bugs by Developer Testing

With Test Driven Development (TDD), you create an executable specification for every method and every class in your code. I have described how to use Test Driven Development (TDD) here.

Using TDD greatly reduces the number of logic errors in your code. It helps you prevent mistakes such as off-by-one errors, null pointer exceptions, division by zero etc.

TDD also gives you a safety net for refactoring your code. The same safety net provides you with a good starting point for finding and fixing bugs in your code, as I describe here.

But TDD alone is not enough. It only proves that your classes work as you intend them to. But it cannot prove that all classes work together. And TDD executes your code in an isolated environment. There might be some surprises when the code is deployed on the target, which might behave differently from what you thought.

This means that TDD is necessary, but you need to do even more testing. Let’s see how.

In software development, you have to make certain assumptions about the system you are programming for. Every processor, every operating system or every browser might behave differently. You have to verify your assumptions about your target before you implement your solution.

One way to do this is to use prototyping. You write a test program that simulates an aspect of the final program on the target.

For example, you might have to write a multithreaded program, but you are not sure how the target behaves. What if you create hundreds of threads? Will there be deadlocks? Will the CPU load rise above all limits? To answer these questions before you decide for a solution, you can write a prototype.

This prototype program could create as many threads as you like. The threads could perform some dummy tasks to simulate system load. You could have them synchronise with each other. Try stopping and starting threads. At the end of the program, see if all threads terminate and if there are no deadlocks. You could even watch the processor load during the execution.

Once you have this prototype, you can use it to verify your assumptions.

Keep your prototype for later use. Maybe you will have to deploy on a different platform, or the number of threads needs to be increased even further. With you prototype at hand, you can check if your multithreading code can stay as it is or if you need to find a new solution.

The most important test for every software is not that it passes all unit tests, but that it works for the user. This means you have to try the software on the target system, the same way as a user would. Only this can prove if you implemented the right thing.

You should test every change you make on the target system. As a consequence, you need to be able to compile the complete system on your own computer. And you have to be able to deploy and run it on the target.

For instance, in the automobile industry the target system of an embedded program is the electronic control unit (ECU). Every software developer needs to test his or her software changes on a real ECU. He needs to flash the changes onto the ECU and to stimulate the input signals to see if the ECU provides the correct output signals. A developer working on a program for Windows has to run the software and use it through its graphical user interface.

In every system that you work on, you have to be able to perform these steps yourself. So before you begin coding, make sure you can try out your own software on the target. With this preparation, you will be able to verify your assumptions about the target system quickly.

When testing your code at the target, also try debugging your software. Even if you don’t find any problems during your tests, let the code run in the debugger and watch it work. If you practice this regularly during development, you will be quicker when you need to find bugs that the customer reports to you.

If your test team already has a test suite for the functionality you are developing, run it to check if the tests are still running. Also run all the tests that might be affected by your changes.

Some parts of the software’s behaviour cannot be observed from the outside. There might be threads that are never terminated or memory that is never released. No test case can notice this just looking at the outputs of the program.

These kind of problems will lead to sporadic and hard to detect errors. To find them, you can log the parameters of the software, such as the number of running threads or the memory consumption.

After the test team ran their tests, check these logs. Did the number of threads rise beyond the limits that you expected? Then try to find out why.

Shipping high quality software is not only a question of knowing your programming language and design patterns well. You also have to test it yourself to ship high quality software.

Write your code test-driven. Use prototyping to verify your assumptions about the target. Always try out the software on the target yourself. Add logging output to your software and monitor it during the automated tests to find hidden bugs.

Use these techniques to deliver software with as few bugs as possible. It will save you a lot of trouble and improve your reputation as a reliable software developer.