TestQuest Inc.

With respect to the development of wireless devices, few investment opportunities offer a more reliable and tangible return on investment (ROI) than test automation.

The last few years have seen worldwide mobile communications advance from concept to reality, and as a result, the number of wireless devices being manufactured to meet the unique needs of businesses and general consumers has skyrocketed. These demands have caused organizations to focus new attention on testing — a crucial step in the product development process.

Wireless devices pose enormous challenges to developers from a product test and verification perspective. They typically have complex user interfaces and displays — such as jog dials, handwriting recognition, multi-level menus, IR ports, audio input and output, etc. — and the software that runs these devices is usually developed on a separate platform.

As a result, organizations have had to rely on manually exercising devices through all of their usage scenarios to recreate the end user experience and validate that wireless data is appearing on the devices as intended. For most organizations, this type of testing involves pulling engineers away from product development to physically punch the buttons on the device being tested, and observe the results - a time-intensive and costly process during which product quality sometimes suffers due to human error.

Test automation has recently emerged as a way to attack this inefficiency. By using testing tools that mimic the actions of a test engineer, companies are finding they can reduce test cycles and time to market, improve product quality and dramatically reduce the overall cost of testing.

The value proposition seems clear, right? But then again, many technology solutions promise increased efficiencies and reduced costs. Wouldn't it be helpful to have a method for assessing your current test processes and identifying the concrete economic benefits of test automation before you invest in a test automation solution for your organization?

This article will discuss ROI as it relates to test automation and provide you with a model for calculating the tangible economic benefits that can be realized through automating the testing process.

Test Automation — The Payoff
The model presented in this article provides an ROI calculation based on the economic benefits test automation offers in terms of reduced cost of testing, reduced time to market and improved product quality. Before we determine your ROI calculation, lets provide some context for each of these three categories:

Reduced Cost of Testing
Probably the most obvious payoff from test automation comes in the form of reduced resource required to conduct the testing process. Testing usually requires an engineer that is familiar with the design of the product, the characteristics of likely or known problem areas, the steps required to recreate a particular state, etc. These people are expensive to the organization, both in terms of the amount they are paid and in terms of the opportunity cost of dedicating them to the highly repetitive task of testing. Reducing these costs are one major advantage of test automation.

How much will automation reduce the cost of testing? This of course varies, but consider these relatively reliable factors:

•Automated testing is faster than manual testing. Our experience shows that automation can easily execute a given test scenario from 25 percent to 50 percent faster than the same tests conducted by humans.

•Automation can execute tests 24 hours a day, seven days a week. With manual tests, the tester is involved throughout the process, whereas with an automated test, the tester only needs to start the script and evaluate the results. This means that organizations that currently use a single 8-hour shift for their test cycles will immediately realize a 67 percent reduction in elapsed time by going to 24 hour a day testing. Factor in weekends and holidays, and the benefits grow larger.

Once an organization has fully automated a given test procedure, the test cycle can be compressed by at least 50 percent, and up to 90 percent. Getting to this point involves the following up-front investments: becoming familiar with the test automation product and creating the scripts that automate the test process. Our experience indicates that most organizations can complete the set-up required to automate a specific test scenario with approximately five times the level of effort involved in one manual test cycle.

Accelerated Time to Market
Every day a product is not available in the market, revenue is reduced by sales being lost to competitors or technology obsolescence. Test automation reduces time to market by compressing the final stage that the product must complete before being delivered to market. What is this reduction worth to the organization? The value reflects net income recaptured by delivering the product to market faster. To put it into perspective, consider the cost to your organization of even one week of lost production. For most organizations, automating the testing process will easily free-up one week of production per year, and will frequently add many more.

Reduced Cost of Quality
Although "quality" can be an abstract concept, the "cost of quality" is highly tangible, and in most cases represents a trade-off that every organization makes, consciously or otherwise. Organizations that skimp on testing do save money on their test expenses, but these are at least partially offset by the need to fix defects once the product is released to market.

In this article, we will use a simple cost of quality metric developed by Rex Black, author of Managing the Testing Process and a well-known consultant specializing in test practice and theory. In his presentation, "Tips for Successful Investing...In Software Testing," Black shows that a key benefit of improved testing is in identifying defects earlier: defects that are found before the product is released to the market are less expensive to fix than defects found after the product reaches the market. Using this approach, Black formulates a "cost of quality" metric that varies with the share of defects found in each stage of the product life-cycle:

Cost of Quality = ((number of defects found prior to release) × (cost of fixing defects prior to release)) + ((number of defects found after release) × (cost of fixing defects after release))

To put a value on the quality improvement offered by test automation, compare the cost of quality using manual test methods to the cost of quality using automated methods. If automated testing provides more rigorous and disciplined testing, it should uncover a greater number of defects earlier in the cycle, and therefore reduce the "cost of quality" by reducing the cost the organization must bear to fix the defects.

The value of this savings will depend on the number of defects identified before vs. after release, and the cost of fixing defects before vs. after release. As a result, the payoff will be most substantial in situations where:

•Defects are subtle or difficult to identify using manual techniques.

•The cost of fixing a defect after the product has reached the market is substantially more than fixing the same defect before the product reaches the market.

Calculating the Payoff
Let's start by considering a single, easily identifiable test situation in your organization. Good candidates include the testing involved in a single product, or the testing involved in a single organization.

First, here are some basic assumptions and definitions to guide your calculation efforts:

Number of test cycles per year: The number of times a complete battery of tests is executed manually from beginning to end.

Number of test engineers: The number of individuals involved in this testing. This need not be individuals that dedicate themselves to testing full-time, but rather get involved in testing as the product goes through its development cycle.

Number of hours spent testing per engineer per cycle: The number of hours, on average, each engineer spends on a single test cycle.

Fully-burdened test engineer cost/year: This includes salary, benefits, overhead, etc. — fully-burdened cost is typically 2 × the engineer's annual salary.

Amortization lifetime: The number of years over which your finance department will amortize your investment in test automation. Most organizations use an amortization lifetime of three years for investments in technology products.

Reduced Cost of Testing Calculation
Hours to create scripts (one time only): How much time will your organization need to create the scripts that automate the test scenario? This usually equals 5 × the total effort involved in a single manual test cycle.

Hours to maintain scripts (each cycle): Some organizations precede each test cycle by tweaking the scripts to add new features, adapt to new requirements, and so on. Although the effort involved is much smaller than the effort required to write the scripts initially, it gets repeated every test cycle, so can be an important figure to consider.

Reduction in engineer's time requirement due to automation: This is a subjective judgment that will be influenced by many factors, but most companies conservatively report a time reduction on the order of 10:1 (or 90 percent reduction). If this sounds high, consider this: an automated test cycle only requires that the engineer start the test script and evaluate the results.

The model shows that automating the test scenario you selected will involve a substantial up-front effort to learn the tool and build the necessary test scripts. Once this work is completed, each test cycle will execute in a small fraction of the time that manual test cycles require. The "Lifetime Savings" figure represents the dollar value of time savings offered by automation over the complete lifetime of the investment in automation. In most situations, this figure is the most reliable, objective and tangible payoff of your investment in test automation.

Reduced Time to Market Calculation
Value of reduced time to market = (Time savings - in weeks) × (unit-production per week) × (margin per unit)

As described above, this figure simply represents the net income generated by the product during the time saved by test automation. As such, we can easily translate the number of hours saved by test automation into a number of weeks available in the market. Factor in the weekly production rate and the margin per unit, and the revenue recaptured becomes an easy calculation.

This figure will usually be surprisingly large: in most ROI analyses, the value of an additional week or two in the market will overshadow the other components in the calculation.

Improved Product Quality Calculation
Number of defects prior to testing: How many defects will be discovered in the product over the course of the product's lifetime covered by this test cycle? This number will be the same regardless of whether or not test automation is used: we are assuming that all of these defects will be found by either your tests or by your customers.

Defect finding efficiency: What share of these defects will be found before vs. after release? The key assumption here is that the increased rigor and repeatability offered by automation will provide at least some improvement here: automated tests will identify some percentage of defects that manual tests would not.

Cost of fixing defects: As discussed above, the model assumes that defects found prior to release are less expensive to fix than defects found after release. Many factors come into play here, such as support costs, costs of issuing patches, and so on.

As described above, this portion of the analysis uses Rex Black's cost-of-quality metric. It supports the idea that the cost of fixing a defect before the product is released to market is less than fixing the same defect after the product is released to market. The analysis includes only the very tangible costs mentioned above, and excludes many less tangible but highly significant costs such as lost customer confidence.

The ROI Calculation
Each of the components evaluated above have produced very tangible, very objective quantification of the savings offered by test automation as it applies to the scenario you selected. From this point, calculating the specific ROI is fairly simple:

ROI = (total benefit ‹ total cost) / (total cost)

Notice that realizing any benefit requires that two major costs be encountered: the investment in the test automation infrastructure and the investment in creating test scripts. As a result, the ROI for the first year will typically be substantially lower than the ROI for subsequent years.

As these calculations illustrate, by automating your test processes, you can reduce test cycles and time to market, improve product quality and dramatically reduce the overall cost of testing for your organization.

For More Information
Visit or call 800.756.1877 (outside North America call +1.952.936.7887).

To obtain a copy of the spreadsheet that accompanies this article please e-mail