What is Software Test Automation?

Developing software to test the software is called test automation. Addresses several problems like:

Automation saves time as software can execute test cases faster than human do.
Test automation can free the test engineers from mundane tasks and make them focus on more creative tasks.
Automated tests can be more reliable.
Automation helps in immediate testing.
Automation can protect an organization against attrition of test engineers.
Test automation opens up opportunities for better utilization of global resources.
Certain types of testing cannot be executed without automation.
Automation means end-to-end, not test execution alone.

Terms used in Automation

Test data generators
Automation should have scripts that produce test data to maximize coverage of permutations and combinations of inputs and expected output for result comparison. They are called test data generators.

Test Suite
When a set of test cases is combined and associated with a set of scenarios, they are called test suite. A Test suite is nothing but a set of test cases that are automated and scenarios that are associated with the test cases.

Skills Needed For Automation

First generation - Record & Playback

Skills for test case automation:

Scripting languages.
Record-playback tools usage.

Second generation - Data Driven

Skills for test case automation:

Scripting languages.
Programming languages.
Knowledge of data generation techniques.
Usage of product under test.

Third generation - Action Driven

Skills for test case automation:

Scripting languages.
Programming languages.
Design and architecture of the product under test.
Usage of the framework.

Skills for framework:

Programming languages.
Design & architecture skills for framework creation.
Generic test requirements for multiple products.

Scope of Automation : What to Automate?

Certain types of testing are more suited for automation, which are:
- Stress, reliability, scalability & performance testing.
- Regression tests.
- Functional tests.
Automate the areas which are less prone to change in future.
Automate the tests which pertain to standards like compliance, legal requirements etc.
Management commitment in Automation
- ROI is to be considered.
- Test cases that are easy to automate should be automated first.

Design & Architecture For Automation

Components in test automation:

External modules.
Scenario & Configuration file modules.
Test cases & Test framework modules.
Tools & Results modules.
Report generator & Report/Metrics modules.

Generic Requirements For Test Tool/Framework

No hard coding in the test suite.
Test case/suite expandability.
Reuse of code for different types of testing, test cases.
Automatic setup and cleanup.
Independent test cases.
Test case dependency.
Insulating test cases during execution.
Coding standards and directory structure.
Selective execution of test cases.
Random execution of test cases.
Parallel execution of test cases.
Looping the test cases.
Grouping of test scenarios.
Test case execution based on previous results.
Remote execution of test cases.
Automatic archival of test data.
Reporting scheme.
Independent of languages.
Portability to different platforms.

Process Model For Automation

The W Model for phases involved in automation.

Selecting a Test Tool

Selecting a test tool is an important aspect of test automation for many reasons:

Free tools are not well supported and get phased out soon.
Developing in-house tools takes time.
Test tools sold by vendors are expensive.
Test tools require strong training.
Test tools generally do not meet all the requirements for automation.
Not all test tools run on all platforms.
Testing tools have very high entry, maintenance, and exit costs and hence careful selection is required.

Criteria For Selecting Test Tools

Meeting Requirements.
Technology Expectations.
Training skills.
Management aspects.

Issues in Selecting a Testing Tool

Meeting Requirements
- Checking wether the tools meet requirements, involves effort and money.
- Test tools are not fully compatible with products.
- Test tools are not tested with the same seriousness as products for new requirements.
- Difficult to isolate problems of product and test suite. Change in product causes test suite to be changed.
Technology Expectations
- Extending the test tool is difficult.
- Requires instrumented code to be removed for certain tests.
- Test tools are not cross platform.
Training skills
- Lack of trainers for test tools.
- Test tools requires people to learn new langauge/scripts.
Management aspects.
- Test tools require system upgrades.
- Migration to other test tools difficult.
- Deploying tool requires huge planning and effort.

Steps for Tool Selection & Deployment

Identify your test suite requirements among the generic requirements discussed. Add other requirements, if any.
Collect the experiences of other organizations which used similar test tools.
Keep a checklist of questions to be asked to the vendors on cost/effort/support/anything else.
Identify list of tools that meet the above requirements.
Give priority for the tool which is available with source code.
Evaluate and shortlist one tool or set of tools and train all test developers on the tool.
Deploy the tool across test teams after training all potential users of the tool.

Summary

A good automation test suite can help in 24x7 test execution, saving effort and time.
The quality requirements for the test suite are equal to or more stringent than that of the product.
At times automation is more complex than product development. Plan to have your best development and test engineers in the automation team.
Selecting test tools without proper analysis will result in expensive test tools gathering dust on the shelf. This is termed as shelf ware.
Automation makes life easier for testers for better reproduction of test results, coverage and, of course, reduction in effort as a side product.

Further Exploration In Different Types of Software Testing
More types of software testing to explore: