Quality Assurance Lessons Learned

Quality assurance planning needs to start when the planning for the project starts. The effort in quality assurance will be 25% to 50% of the project effort. Planning early provides the time to properly prepare the activities and provides insight into the quality of the code early in the project. Some project managers start the quality assurance late in the project, figuring that testing is not needed until the software is nearly complete. With this approach, the project manager can be unaware of problems with the quality of the code. Also, there may not be enough time to perform good automated testing and performance testing.

Developers must take responsibility for the correctness of their code. In some projects, the developers take the attitude that the QA staff will detect any defects introduced during development. Developers may feel that they can save their own time by not performing extensive unit testing, since the QA will test anyway. This process is false economy, however, for several reasons:

• It is impossible for testers to test all aspects of a program, even a small program. There is no guarantee that a tester will find a defect that could be detected through inspection or unit testing.
• Formal testing by the QA staff is relatively labor intensive compared to unit testing. During unit testing, a developer can run a number of test cases, correct defects, and verify them. With formal testing, the tester runs the test case and logs any defects. Finding defects slows down the testing process. The tester usually has to determine how to repeat the generation of the defect. The defect report has to be written. The project manager has to review the defect report and assign correction to the appropriate developer. The developer then has to determine what the best approach is to reproduce the error. He then performs tests, possibly the same tests he would have performed during unit testing. Once the defect is identified, the developer checks in the code. It then must be scheduled for a build and promoted into the test environment. The tester must then reexecute the test case and document the results. If the number of defect reports becomes large enough, the project manager and other managers must spend time prioritizing defects. Instead of a few days of thorough unit testing by a developer, the whole process can drag on for weeks, involving not only the developer, but testers and managers.
• The developers perform white box testing, that is, testing based on understanding the inner workings of the code. Most testing by the QA staff is black box testing, testing against requirements without knowing about the inner workings.

To develop correct code, developers must either understand the requirements or seek clarification.

Test-driven development is one of the most powerful concepts emerging from agile development. See [BECK03]. It is one of the easiest innovations to implement in a development organization. Test driven development is a method for developing software. With this method, automated tests are developed before coding the feature. The developer then codes the features so that all tests are passed.

As development progresses, the development team builds up a set of automated tests. This asset can be used for regression testing as additional changes are introduced into the code. Some advanced development organizations organize these tests as smoke tests that are run daily or more often to insure that changes do not break existing code. Over time, thousands of test are accumulated.

An additional advantage of test-driven development is that the automated tests can be reviewed for completeness. This provides project management with insight into the extent of coverage of the unit tests.

To encourage proper unit testing, it should be easy for developers to perform those tests. With many modern systems, there are a number of interfaces to other systems. Sometimes, the development environment does not include interfaces to these systems. Therefore, developers cannot test the interfaces to those systems. In other cases, developers need extensive test data staged for testing. This is a task with which the QA staff can assist. Without this test data, developers cannot perform adequate testing. Since adequate test data must eventually developed, the project might as well make this data available to the developers.

Traditionally, testing has meant testers key data into the application. Since iterative development means that versions of software are produced at regular intervals, manual testing because time consuming and expensive. Open source tools make it possible to develop automated tests, including tests that operate through the user interface of the application. Automating test cases converts a perishable effort into a reusable asset. Although automated test cases require development and maintenance effort, the ability to rerun the tests repeatedly justifies the investment.

A good scripting language like Ruby or Python has many uses in quality assurance:

• search, correlate, or summarize information in program logs
• perform database queries to check validation or business rules
• perform database queries and calculations to verify the content of report
• generate XML inputs to test interfaces.

In addition to scripting languages, there are several good open source tools to assist in quality assurance:

• Watir for automated testing through the browser
• The Grinder for automated load testing
• xUnit frameworks for unit testing
• Finesse for test organization and documentation.

During the quality assurance activities, statistics are generated and need to be collected, analyzed, and made available to the project team and management. Types of statistics that should be collected include:

• number of test cases by category of test
• number of test cases performed
• effort expended in executing test cases
• number of inspection performed
• number of defects uncovered in inspections
• number of defect reports generated per week by component
• number of defect reports closed per week by component
• total number of defect open by component
• average effort to close a defect report.

If collecting and processing these statistics is not explicitly assigned to a team member, it is likely that this task will not be performed when the workload gets heavy. The project manager or QA manager is unlikely to have the time to perform this task, although the metrics generated will be vital to their jobs.

From the beginning of the project, the project team creates a number of documents. Many documents will have several versions. It is easy for the volume of files to grow so that it is difficult to find the current version of requirements and other documents. Some good practices include:

• Appoint a team member as document librarian. This part time job includes setting up categories for information, reorganizing files and directories, and deleting obsolete files.
• Control documentation through one mechanism. If documents are maintained on a network disk, keep all the documentation there. Do not place some of the documentation on a network disk, some of it on a Web site, and some in a document management system.
• Clear out obsolete versions of files. If there possible reasons to save prior versions of a file, create a subdiectory labeled "obsolete" or similar name to store the older files.
• Consider using a source code control system like Subersion to manage documentation.
• Where possible, maintain a revision history for each document.
• Establish a mechanism other than e-mail for distributing documents. E-mail requires each recipient to take the time to organize and save documents. With a central repository, documents management is more efficient.

A good defect tracking tool greatly improves the management of defect reports. The number of defects can easily overwhelm a spreadsheet based system. Also, spreadsheets are difficult to share among several people. There are several good systems. Consider Jira.