What is Regression Testing?

Regression testing is done to ensure that changes work as designed and do not have any unintended side-effects.
Regression tests acknowledge that new fixes can cause new 'side effects' and can also cause some older defects to appear.
The challenge in designing and running regression tests centers around designing the right tests to combat new defects introduced by the immunity acquired by a program against old test cases.
Regression testing is done to ensure that enhancements or defect fixes made to the software works properly and does not affect the existing functionality.
Regression testing is important in today's context since software is being released very often to keep up with the competition and increasing customer awareness.
It is essential to make quick and frequent releases and also deliver stable software.
Regression testing enables that any new feature introduced to the existing product does not adversely affect the current functionality.
Regression testing follows selective re-testing technique.
Whenever the defect fixes are done, a set of test cases that need to be run to verify the defect fixes are selected by the test team.
An impact analysis is done to find out what areas may get impacted due to those defect fixes.
Based on the impact analysis, some more test cases are selected to take care of the impacted areas.
Since this testing technique focuses on reuse of existing test cases that have already been executed, the technique is called selective re-testing.

Types of Regression Testing

What is a build?
A build is an aggregation of all the defect fixes and features that are present in the product.

Two types of regression testing:

Regular regression testing.
Final regression testing.

Regular Regression Testing

Regular regression testing is done between test cycles to ensure that defect fixes done and functionality that was working with the earlier test cycles continues to work normally.
Regular regression testing can use more than one product build for the test cases to be executed.

Final Regression Testing

A “final regression testing” is done to validate the final build before release.
The Configuration Management(CM) engineer delivers the final build with the media and other contents exactly as it would go to the customer.
The test cycle is conducted for a specific period of time, which is called cook time.
Cook time is necessary as some defects show up only after running the product build for some time.
The final regression test cycle is more critical than any other type or phase of testing, as the build which is being tested is the one which will go directly to the customer.

When to do Regression Testing?

Perform regression testing when:

A reasonable amount of initial testing is already carried out.
A good number of defects have been fixed.
Defect fixes that can produce side-effects are taken care of.
Regression testing may also be performed periodically, as a pro-active measure.

How to do Regression Testing?

A well-defined methodology for regression testing is very important as this among is the final type of testing that is normally performed just before release. If regression testing is not done right, it will enable the defects to seep through and may result in customers facing some serious issues not found by test teams.

Performing an initial “Smoke” or “Sanity” test
Understanding the criteria for selecting the test cases
Classifying the test cases into different priorities
A methodology for selecting test cases
Resetting the test cases for test execution
Concluding the results of a regression cycle

Initial 'Smoke' or 'Sanity' Test

Smoke testing consists of:

Identifying the basic functionality that a product must satisfy.
Designing test cases to ensure that these basic functionality work and packaging them into a smoke test suite.
Ensuring that every time a product is built, this suite is run successfully before anything else is run.
If this suite fails, escalating to the developers to identify the changes and perhaps change or roll back the changes to a state where the smoke test suite succeeds.

Criteria for Selecting the Test Cases

Criteria to select test cases for regression testing are as follows:

Include test cases that have produced the maximum defects in the past.
Include test cases for a functionality in which a change has been made.
Include test cases in which problems are reported.
Include test cases that test the basic functionality or the core features of the product which are mandatory requirements of the customer.
Include test cases that test the end-to-end behavior of the application or the product.
Include test cases to test the positive test conditions.
Includes the area which is highly visible to the users.

Classifying Test Cases

Priority-0

These test cases can be called sanity test cases which check basic functionality and are run for accepting the build for further testing.
They are also run when a product goes through a major change.
These test cases deliver a very high project value to both to product development teams and to the customers.

Priority-1

Uses the basic and normal setup and these test cases deliver high project value to both development team and to customers.

Priority-2

These test cases deliver moderate project value.
They are executed as part of the testing cycle and selected for regression testing on a need basis.

Methodology for Selecting Test Cases

This methodology takes into account the criticality and impact of defect fixes after test cases are classified into several priorities:

Case 1

If the criticality and impact of the defect fixes are low, then it is enough that a test engineer selects a few test cases from test case database (TCDB), and executes them.
These test cases can fall under any priority (0, 1, or 2).

Case 2

If the criticality and the impact of the defect fixes are medium, then we need to execute all Priority-0 and Priority-1 test cases.
If defect fixes need additional test cases from Priority-2, then those test cases can also be selected and used for regression testing.
Selecting Priority-2 test cases in this case is desirable but not necessary.

Case 3

If the criticality and impact of the defect fixes are high, then we need to execute all Priority-0, Priority-1 and a carefully selected subset of Priority-2 test cases.

Some Other Methodologies

Regress all

All priority 0, 1, and 2 test cases are rerun.
This means all the test cases in the regression test bed/suite are executed.

Priority based regression

All priority 0, 1, and 2 test cases are run in order, based on the availability of time.
Deciding when to stop the regression testing is based on the availability of time.

Regress changes

Code changes are compared to the last cycle of testing and test cases are selected based on their impact on the code.

Random regression

Random test cases are selected and executed.

Context based dynamic regression

A few Priority-0 test cases are selected, and based on the context created by the analysis of those test cases after the execution and outcome, additional related cases are selected for continuing the regression testing.

Resetting the Test Cases for Regression Testing

Resetting of test cases is not done often, but if done, these points should be kept in mind:

When there is a major change in the product.
When there is a change in the build procedure which affects the product.
Large release cycle where some test cases were not executed for a long time.
When the prodcut is in the final regression test cycle with a few selected test cases.
Where there is a situation, that the expected results of the test cases could be quite different from the previous cycles.
The test cases relating to defect fixes and production problems need to be evaluated release after release.In case they are found to be working fine, they can be reset.
Whenever existing application functionality is removed, the related test cases can be reset.
Test cases that consistently produce a positive result can be removed.
Test cases relating to a few negative test conditions, which are not producing any defects, can be removed.

Concluding the Results of Regression Testing

Result 1
If the result of a particular test case was a pass using the previous builds and a fail in the current build, then regression has failed. A new build is required and the testing must start from scratch after resetting the test cases.

Result 2
If the result of a particular test case was a fail using the previous builds and a pass in the current build, then it is safe to assume the defect fixes worked.

Result 3
If the result of a particular test case was a fail using the previous builds and a fail in the current build and if there are no defect fixes for this particular test case, it may mean that the result of this test case should not be considered for the pass percentage. This may also mean that such test cases should not be selected for regression.

Result 4
If the result of a particular test case is a fail using the previous builds but works with a documented workaround and if you are satisfied with the workaround, then it should considered as a pass for both the system test cycle and regression test cycle.

Result 5
If you are not satisfied with the workaround, then it should be considered as a fail for a system test cycle but may be considered as a pass for regression test cycle.