Testing Is To Find Errors
The 1st principle is “Testing is the process of executing a program with the intent of finding errors”. The 1st edition has “finding an error” rather than “finding errors”.
The problem people have with this principle is they believe that testing is about showing that a program or system works. They adopt a system of belief in the system rather than being properly sceptical and adversarial. They think that just running a few tests, they believe will work, are sufficient. Whereas they need to run tests designed to make the system fail. Only by doing are they doing useful testing.
It follows from this principle that you do not stop testing just because the tests are failing. This is what you are seeking to and therefore you continue running tests. The only reason to stop is if the system is so badly fouled up that no useful tests can be run but this should be a very rare decision.
Tests Are Designed To Find New Errors
The 2nd principle is “A good test case is one that has a high probability of detecting an as yet undiscovered error”. The 1st edition has “finding” rather than “detecting”.
The focus of this principle is the design of test cases. The aim should be to focus on trying to find new errors in a system and not on finding the same error again. It has two important aspects:
- It is a restatement of the Equivalence Partition principle of finding areas of the system which behave the same and testing them once. If they fail it is deemed all that area has failed, if it passes then all have passed.
- It is also guides the development of test cases in those areas that are most likely to fail. These are:
- Areas which have already have had faults and
- Areas which have been extensively modified.
The key is to design the test cases to give the biggest opportunity for them to fail.
Successful Tests Find new Errors
The 3rd principle is “A successful test case is one that detects an as yet undiscovered error”. The 1st edition has “uncovers” rather than “detects”.
This principle deals with running the test cases and the reaction the tester has to the results. Running a test does not detect an error by itself. All it can do is show is a difference between the Expected Results from the test case and the Actual Results. This difference can then be evaluated to see if an error exists in the system. Therefore when a difference is found the reaction of the tester is joy at having found a potential error. If the expected and actual results are the same, indicating the test has passed, then this is an unsuccessful test case and the tester is unhappy.
The Happy Tester
These principles lead you the concept of happy testers. These are people who are happy when test cases fail in new and unique ways because it will allow them to find an error, and they are unhappy when tests pass because they have found no new errors.
A happy tester strives hard to come up with a list of unique errors with a program or system using the minimal number of tests to do the job.