The Changing Nature of Quality Assurance

Quality assurance is an indispensable part of any software project. However, it has drawbacks which any project must try to mitigate.

Testing cannot be totally successful

It is impossible to test all aspects of even a small program. Even all interesting test cases cannot be run as often as desired.

Testing is expensive and time consuming

Quality assurance can consume 25% to 50% of project effort

Quality assurance is documentation intensive

Typically, the quality assurance effort involves these types of documentation:

• Requirements
• Planning
• Test cases

This documentation costs time and money.

Extensive manual effort

Much of the quality assurance effort requires manual keying of data. This effort tends to be "perishable." Once the code is changed, much of this effort needs to be repeated.

Late in the process

Often most testing is done late in the process. Therefore, defects are discovered near the end of the project. This can be a cause of unexpected project delays.

Quality assurance automation could require expensive tools and may be ineffective

In the past, many automated test tools cost thousands of dollars. In addition, tools drive other costs like training expense, learning curve, and maintenance fees.

With iterative development practices, like extreme programming, becoming more common, automated testing becomes increasingly important. Manual testing through the user interface becomes increasingly more expensive and less effective. Since code changes are more frequent than previously, regression testing has to be done more often. In practice, either regression testing is undercut or regression testing costs increase. Automated testing is a means of converting a perishable manual activity into a reusable asset.

Problems with Automated Testing in the Past

In the past, automated testing meant recording and playing back scripts. For good coverage, large number of scripts had to be recorded. Scripts broke if user the interface changed. As a result, there was a tendency to initiate automated testing toward end of project where it was less effective. In addition, updating the scripts required considerable effort.

The Solution

Automate through programming interfaces when possible

In many cases, automated tests can perform their work by calling methods in the programs. This approach has advantages over testing through the user interface. The user interface tends to change more frequently than the underlying programming interfaces. Therefore, automated tests that use programming interfaces tend to require fewer updates than tests through the user interface.

Use modular user interface testing

If automated tests are built by recording a series of steps using a user interface recording tool, then any changes to the user interface require recording the scripts or manually modifying all the scripts affected. A better approach is to build a series of modules, where each module handles part of the application. For example, each module could handle a particular screen. A change in a screen involves changing one module, rather than a set of scripts.

Use open source tools

There are a number of open source tools that can be used in lieu of expensive proprietary tools.

• Watir, automated testing of Web applications through the browser.
• The Grinder, performance testing of Web applications.
• Finesse, for test organization and documentation management

Build automated tests throughout the project

If the QA team begins building automated tests when the application is first developed, there will probably be enough time to build a set of test cases with substantial coverage of the applications. The automated tests can then be run as changes are introduced into the application as shakedown and regression tests.

Use FIT test approach

FIT is both a methodology for building tests and a framework for executing automated tests. With FIT, tests are designed as tables of inputs and expected results. Business analysts prepare the tables, based on their knowledge of the domain. Programmers writes fixtures the implement the automated test cases.

FIT helps users and business analyst clarify requirements. It helps developers understand requirements. Test cases can be written rapidly. Execution of the tests are automated.

Performance testing involves determining if the application can support the required number of users and operate for the required period of time. Performance testing requires automation. It is almost impossible to obtain the required load levels and consistency in testing through manual methods. It also requires a substantial amount of planning. It requires development work, so enough time has to be allocated to permit development of the tests, repeated execution of the tests, analysis of the results, and remediation of any problems. Commercial tools can be expensive. The Grinder provides an open-source solution for performance testing of Web-based applications.

Start QA planning when project planning starts

Quality assurance needs to start when the project starts. It is a mistake to delay quality assurance until the software is nearly finished. Quality assurance is the eyes and ears for the project manager, sponsors, and users. If QA is not functioning during the full project, these stakeholders will be blind with regard to the quality of the code.

Prepare a Software Quality Assurance Plan

If your organization has a template for a Software Quality Assurance Plan, use it. Otherwise, don't develop new one. Use the Software Quality Assurance Plan based on IEEE Standard 730-2002. This IEEE Standard covers the necessary aspects of QA. The plan should be a living document. Keep it up-to-date as new information becomes available.

Maintain an integrated schedule

The project manager and the QA manager should integrate the schedule for quality assurance activities with the rest of the project schedule. Avoid replication of schedule information in various documents, spreadsheets, and MS Project files.

FIT

FIT is both a methodology and a software framework for developing test cases.

Ruby

Ruby is an object-oriented scripting language that can be used for a variety tasks. Watir, a tool for automated testing of Web applications, is written in Ruby. Ruby can be used for scripts to access and test data in a database. It can also be used for analyzing program logs.

The Grinder

The Grinder is an open source tool for performance testing. It offers a generic approach to performance testing. The approach includes common cases such as HTTP web servers, SOAP and REST web services, and application servers (CORBA, RMI, JMS, EJBs), as well as custom protocols. Flexible scripting tests are written in the powerful Jython scripting language. A graphical console allows multiple load injectors to be monitored and controlled, and provides centralized script editing and distribution. Mature HTTP support includes automatic management of client connections and cookies, SSL, and proxy awareness. Tools provide recording and replay of the interaction between a browser and a Web site.

What is Model Driven Testing?

Model Driven Testing is the use of a formal, executable model of the software requirements to test the software. The requirements are defined in a computer language suitable for specifications. The model is then programmed to generate a set of test cases. For example, the model could generate a set of FIT tables.

Model Driven Testing is appropriate for complex requirements where precision is needed. For example,

• business rules
• calculations
• data structures
• data transformations
• workflows

Validation of the Model

A formal model can be verified is several ways:

• Inspection against textual requirements
• Comparison of values with prior systems
• Comparison of values with client-supplied spreadsheets
• Comparison of values with the system under development.

Advantages of Model Driven Testing

• Clarifies complex requirements
• Can generate a large number of test cases quickly

• QA planning and activities need to start at begin of project.
• QA testing needs to be automated.
• Requirements process needs to supply the necessary documentation.
• Use FIT methodology for developing test cases.
• Open Source tools can aid in adopting test automation.
• Model based requirements and testing should be used for complex rules and calculations.