INTRODUCTION:
As stated in my previous post, safety-critical software is expensive to develop and static analysis tools are highly recommended by both certification standards and practitioners in the field. Even more expensive than developing software is the result of software failures, from recalls to litigation to damaged reputation. Toyota’s unintended acceleration flaw, for example, in its drive-by-wire throttle system, cost the company up to $5 billion dollars in damages and lost revenues. A significant loss and important lesson in software safety.
This post looks into more detail on the ROI for using static analysis tools in safety-critical product development. In particular, static analysis has a role in each applicable phase of the V-model of software development, often referenced in safety standards and certifications.
Related:
- Safety critical software and development productivity
- Accounting for Toyota’s Recalls
- The Economic Impacts of Inadequate Infrastructure for Software Testing
- Static Analysis Essential for Affordable Safety Critical Software
The Exponential Cost of Failure
Although the Toyota example is an extreme, a significant software failure can have almost unbounded financial impact — studies have shown that defects cost 100 times more to fix in production than in early phases of development.
As pointed out by Capers Jones (2009), looking at a cost-per-defect metric alone is misleading since it doesn’t factor in the volume of defects and the fact that the cost to find and repair a defect is often the same over time (something developers are quick to point out). For safety-critical embedded systems, however, the cost of repair is higher than other industries. If a safety-critical defect isn’t fixed on time (or worse, purposely hidden), the financial impact can escalate to legal liability and impact of future revenue.
Considering the typical software development lifecycle illustrated by the V-model below, we can consider the relative benefits of static analysis at each phase of development. The V-model is a good example here, being featured in many of the safety critical software certification standards (e.g. IEC 61508 and ISO 26262).
Figure 1: The V-model of system development, often referenced in safety critical standards such as IISO 26262.
Rather than looking at the traditional cost-per-defect over time or per phase, which Jones (2009) argues is true mathematically but doesn’t reflect what is seen in practice, the more revealing data is the cost-per-defect per relative volume of defects (as volume of defects decreases over time and each phase of development).
Figure 2: The total cost and cost per defect as the volume of defects diminishes. Source: Capers Jones (2009)
What is interesting about Figure 2 (note the development phase order), is that the cost per defect at each phase goes go up as expected, but total costs are going down due to the decreased volume of defects. In practice, it doesn’t take longer to find and fix bugs at each phase, but the costs are still there despite diminished volume. It’s worth noting, also, that as a product matures into operation and maintenance (not covered in the chart), cost-per-defect is much higher due to the impact of servicing a fielded product. The other intangible costs such as damage to brand and loss of future customers and income, are still factors to consider.
The Return on Investment for Static Analysis
Static analysis tools are highly recommended in software safety standards and rightfully so. Finding defects early is still a big cost saver because, as seen in figure 2, it’s where a majority of the costs lie. Static analysis helps reduce cost, time, and money in the following ways:
- Finds defects before unit testing: Static analysis tools can be used right at the developer’s desktop environment and can prevent defects before they enter the build system and the unit test phase of development.
- Finds defects that testing misses: Unit testing, even on projects demanding high code coverage levels, can still miss important defects.
- Prevents defects in the first place: Enforcing strict coding standards, such as MISRA C, can help prevent many classes of defects in code. Enforcing good discipline in coding and creating a develop-analyze-test micro cycle for small code changes can prevent many defects from being created in the first place.
So what is the return on investment given these factors? Static analysis decreases the volume of defects in software under development at all stages of development. A simple analysis is to reduce the number of defects from the data we have from Figure 2. Given this reduction in created defects during development, we can see a significant reduction in cost:
Figure 3: The savings versus total cost of reducing the number of defects entering testing at each phase by 25%.
This simple analysis yields about $126 savings per defect, which, given an average of 15 defects per 1000 line of code (during development when defect volumes are high), gives a savings of $1,900 per 1000 LOC. Of course, results will vary, also based on other factors such as labor rates, defect detection and repair time, and defect density. However, given that many safety-critical systems employ 100 KLOC or more, the business case for static analysis is clear. (This analysis also doesn’t include post-deployment costs which, as stated before, are much higher.)
Static Analysis is More Than Just Defect Reduction
In addition to defect-detection, CodeSonar is used to detect complex concurrency issues, analyze third-party source and binaries, and detect errors that traditional testing misses. These critical benefits are not factored into the rather simple analysis above, but clearly add to the tool’s ROI. However, finding defects that “slip through the cracks” give the greatest economic benefits to the development team.
CONCLUSION:
Failures in safety-critical software have catastrophic effects in human and economic terms. Static analysis tools are highly recommended for safety-critical software, to ensure the development of software that is secure and high-quality. In some cases, safety certification standards require static analysis tools because of their ability to find defects that testing may miss and to enforce coding standards (among other benefits). The return on investment for static analysis tools is compelling, underscoring that static analysis plays an important part during development but also in system deployment into the marketplace.
