Best Practices in Regression Testing

Practice 1: Regression can be used for all types of releases

Regression testing can be applied when

• We need to measure the quality of product between test cycles (both planned & need based);
• We are doing a major release of a product, have executed all test cycles, and are planning a regression test cycle for defect fixes; and
• We are doing a minor release of a product (support packs, patches, and so on) having only defect fixes, and we can plan for regression test cycles to take care of those defect fixes.
• There can be multiple cycles of regression testing that can be planned for every release. This applies if defect fixes come in phase or to take care of some defect fixes not working with a specific build.

Practice 2: Mapping defect identifiers with test cases improves regression quality

When assigning a fail result to a test case during test execution, it is a good practice to enter the defect identifier(s) (from the defect tracking system along, so that you will know what test cases are to be executed) when a defect fix arrives. Please note that there can be multiple defects that can come out of a particular test case and a particular defect can affect more than one test case.

Even though ideally one would like to have a mapping between test cases and defects, the choice of test cases that are to be executed for taking care of side effects of defect fixes may still remain largely a manual process as this requires knowledge of the interdependences amongst the various defect fixes.

As the time passes by and with each release of the product, the size of the regression test cases to be executed grows. It has been found that some of the defects reported by customers in the past were due to last-minute defect fixes creating side effects. Hence, selecting the test case for regression testing is really an art and not that easy. To add to this complexity, most people want maximum returns with minimum investment on regression testing.

Practice 3: Create and execute regression test bed daily

To solve this problem, as and when there are changes made to a product, regression test cases are added or removed from an existing suite of test cases. This suite of test cases, called regression suite or regression test bed, is run when a new change is introduced to an application or a product. The automated test cases in the regression test bed can be executed along with nightly builds to ensure that the quality of the product is maintained during product development phases.

It was mentioned earlier that the knowledge of defect, product, their interdependences and a well-structured methodology are all very important to select test cases. These points stress the need for selecting the right person for the right job. The most experienced person in the team or the most talented person in the team may do a much better job of selecting the right test cases for regression than someone with less experience. Experience and talent can bring in knowledge of fragile areas in the product and impact the analysis of defects.

Practice 4: Ask your best test engineer to select the test cases

Strategy 1: The tiger has been put in a cage to prevent harm to human kind
Strategy 2: Some members of a family  lie inside the mosquito net as prevention
against mosquitoes.

Strategy1 has to be adopted for regression. Like the tiger in the cage, all defects in the product have to be identified and fixed. This is what “detecting defects in your product” means.

Strategy2 signifies “protecting your product from defects”. The strategy followed here is of prevention.

Practice 5: Detect defects, and protect your product from defects and defect fixes

Another aspect relating to regression testing is “protecting your product from defect fixes”. As discussed earlier, a defect that is classified as a minor defect may create a major impact on the product when it gets fixed into the code. It is similar to what a mosquito can do to humans (impact), even though its size is small. Hence, it is a good practice to analyze the impact of defect fixes, irrespective of size and criticality, before they are incorporated into the code. The analysis of an impact due to defect fixes is difficult due to lack of time and the complex nature of the product. Hence, it is a good practice to limit the amount of changes in the product when close to the release date. This will prevent the product from defects that may seep in through the defect fixes route, just as mosquitoes can get into the mosquito net through a small hole there. If you make a hole for a mosquito to get out of the net, it also opens the doors for new mosquitoes to come into the net. Fixing a problem without analyzing the impact can introduced a large number of defects in the product. Hence, it is important to insulate the product from defects as well as defect fixes.

If defects are detected and the product is protected from defects and defect fixes, then regression testing become effective and efficient. Regression testing, in effect, provides the mosquito net.