How a Test Framework Runs the Tests
To write a new test, you implement a test method in the same language as the class that you are about to test. A test is just a method itself, which has a little extra information to tell the test framework that it is a test.
Then you run the test framework. This can usually be done on the command line, but many IDEs such as Visual Studio also allow you to run the tests directly from the IDE.
The test framework checks all source files in a directory for tests. It puts all the tests it finds into a list. After that, it executes all tests in the list and checks their results. At the end, it will present you a summary of the test execution.
So far, so good. But how does the framework know which methods are tests? You have to mark the methods as tests. How to do this depends on the programming language and the test framework.
For example, the C++ test framework GoogleTest uses macros:
TEST_F(MathematicalExpressionParserTest, testAddition)
{
...
}
The Java framework JUnit uses annotations. These are special properties that you can assign to methods. The Java Runtime Environment will read them and know that these methods contain tests.
class MathematicalExpressionParserTest
{
@Test
void testAddition()
{
....
}
}
And Python recognizes methods as test methods, if their names start with test:
def test_addition():
...
So by simply giving a method the appropriate name or using an annotation, you can create a unit test.
Test Fixtures
Tests that belong together can be grouped into a test fixture. The test fixture is a class that implements an interface provided by the test framework to mark it as a test fixture. The test fixture class can contain several test methods. When executing a test fixture, the test fixture will execute all tests inside this class and group them in the result.
The real power of a test fixture is that you can write code that runs before and after each test is executed. These methods are often called Setup and Tear Down or similar. You can put code that is needed for every test into a Setup method. Examples for tasks that need to be executed before a test runs are the creation of complex objects or the acquisition of a database lock.
After the Set Up method has finished, the test framework runs the test and checks the result.
Following the completion of the test method, the framework will run the Tear Down method. The Tear Down method is responsible for leaving the test environment in a clean state. It might reset a test database that is used by all tests, or it might cleanup objects that cannot be destroyed by the garbage collector.
It is very important that each test cleans up after it was executed. If this does not happen properly, the tests might behave differently when the order of the test changes. This will cause you much trouble debugging your tests.
As an example of how a test fixture is executed, look at this test fixture class in Python:
import unittest
class TestClass(unittest.TestCase):
def setUp(self):
print("setUp")
def tearDown(self):
print("tearDown")
def test_1(self):
print("Test 1")
def test_2(self):
print("Test 2")
def test_3(self):
print("Test 3")
if __name__ == '__main__':
unittest.main()
It contains three test methods and additionally one method setUp and one method tearDown. When you run this test fixture, it produces the following output. The output shows that setUp and tearDown are called before and after each test method:
setUp Test 1 tearDown .setUp Test 2 tearDown .setUp Test 3 tearDown .
Use test fixtures extensively so you don’t have to write code for the preparation and cleanup work in each of your tests separately. This also ensures that your tests start with the correct state, so they can run independently from each other.
How to Check for Success
The test framework not only runs the test, it also checks for success or failure of the tests. But what does success or failure mean? You have to tell the test framework how to perform these checks.
The framework provides methods that check for boolean conditions. If these conditions are not met, then the test framework will mark the test case as failed and print out the kind of error. These methods are called asserts, and they come in different flavors.
The simplest version is assertTrue. It checks that a boolean condition is true. If it is not true, the test fails.
Then there are more specific versions like assertGreaterEqual, which checks if one expression is equal to or greater than another one. The more specific versions make the code easier to read, and they also make the error output of the test more informative.
Consider this example. Both of the following tests perform the same check:
def test_addition_1(self):
sum = 3 + 5
self.assertTrue(sum == 8)
def test_addition_2(self):
sum = 3 + 5
self.assertEqual(sum, 8)
But if we make them fail by changing the expected value to 7, then the second version produces the more comprehensible output:
======================================================================
FAIL: test_addition_1 (__main__.TestClass)
----------------------------------------------------------------------
Traceback (most recent call last):
File "testFixtureWithAsserts.py", line 7, in test_addition_1
self.assertTrue(sum == 7)
AssertionError: False is not true
======================================================================
FAIL: test_addition_2 (__main__.TestClass)
----------------------------------------------------------------------
Traceback (most recent call last):
File "testFixtureWithAsserts.py", line 11, in test_addition_2
self.assertEqual(sum, 7)
AssertionError: 8 != 7
----------------------------------------------------------------------
Ran 2 tests in 0.000s
Therefore, it is always good to use the most specific version of every assertion, because it makes it much easier to determine the reason for a failed test.
You can make the error message even more specific by providing a custom error message.
def test_addition_3(self):
sum = 3 + 4
self.assertEqual(sum, 8, "3 + 5 should equal 8")
This produces the following output:
======================================================================
FAIL: test_addition_3 (__main__.TestClass)
----------------------------------------------------------------------
Traceback (most recent call last):
File "testFixtureWithAsserts.py", line 15, in test_addition_3
self.assertEqual(sum, 8, "3 + 5 should equal 8")
AssertionError: 3 + 5 should equal 8
Try to always make the error message as meaningful as possible. This will help you greatly to locate the error whenever a test fails. The best way to achieve this is to check the error message when the test fails the first time. This is the RED stage of the TDD cycle. Look at the error message that you receive. If it is not understandable, improve it now. This will improve the quality of your tests and will help you and your colleagues in the future.
Assertions and Expectations
Some test frameworks support not only assertions, but also expectations. Both can be used in almost the same way. The difference between them is that an assertion aborts the execution of the test as soon as it fails, and the expectation lets the test continue.
If your test framework supports expectations, you can write tests that check a necessary condition with an assert. If this condition is not true, it does not make sense to continue with the test. A typical check using an assertion is that a reference is valid. If it is not valid, no further checks can be done on it, so the test must be aborted. But if it is valid, several different properties of the reference can be checked with expectations. Thus the test result can show more than one failure on the properties at once.
Learn to distinguish between assertions and expectations to make your tests more readable and the results easier to understand.
Outlook
A test framework is a necessary tool for doing test driven development. Many different frameworks are available for almost all programming languages.
To start test driven development, make yourself familiar with a test framework for your programming language. You can find an overview of test frameworks in this article. Once you have mastered the basics, you will easily understand other test frameworks as well.
One reply on “How to Use Test Frameworks”
[…] you need to learn how a test framework works. In addition to that, will have to learn about Test Doubles. To bring all of this together, […]