Calculating the ROI of SAST in DevSecOps for Embedded Software

With the increasing reliance on software driving critical functionality in all types of products such as industrial controls, medical devices, automotive sensors, flight control systems and so much more, ensuring the quality, safety and security of the software is more essential than ever. To achieve these results, continuous testing is necessary, but can be perceived as costly. Choosing the right Static Application Security Testing (SAST) solution can enhance software development processes, provide significant efficiencies and deliver high-quality products, while offering a very attractive ROI.    

Get Started by Embracing DevSecOps
DevSecOps (Development-Security-Operations) improves the DevOps (Development-Operations) pipeline to where security is a critical part of the development process. The realization here is that a security failure is the same, or worse, as a quality failure. Security is a differentiator but not at the expense of innovation and time to market. Software organizations don’t intentionally leave out security but unless it’s part of the development culture, it doesn’t get done. At huge risk, software teams are delegating security until the end of the development cycle. Unfortunately, ‘tacking on’ security at the end can lead to inefficiencies, cost overruns, project delays and vulnerabilities in the final product

Making security part of an organizational culture and DevOps pipeline requires careful planning, training, expertise and the right automation support. To reduce the impact of DevSecOps, integrating the right tools into the security and quality process is the key.

In addition, a key reason to build security into agile and continuous processes is to build upon the knowledge that accumulates over the project. It’s not reasonable to expect software teams to understand their complete attack surface, for example, at the beginning of the project. Building security into day-to-day operations accumulates expertise and knowledge. Starting early is the key. This is often referred to as “shifting left” or introducing security into the development phases of the SDLC.

It’s through this push for improving security throughout the lifecycle, as early as possible that DevSecOps achieves its return on investment (ROI). For every design flaw that is caught before implementation, every software weakness that is removed before it becomes a vulnerability and every vulnerability that is removed before integration and testing, cost saving becomes much larger than if a product is released with serious security vulnerabilities. Simply put, a dollar spent on security improvement now saves hundreds of thousands later in the software lifecycle. How do we get to such a number?

The Simple Answer: ROI for Early Defect and Vulnerability Removal with SAST

Static application security testing (SAST) solutions are highly recommended in software safety standards and in DevSecOps, rightfully so. Finding security vulnerabilities and defects early is a huge cost saver because it’s where a majority of bugs are introduced. SAST helps reduce cost, time, and resources in the following ways:

• Finds defects before testing: SAST solutions can be used right at the developer’s desktop and tools? environment to identify defects while coding and can prevent these defects before they enter the build system and later stages of development. Every defect removed at this point saves the team from failed unit, integration and system tests plus additional debugging and retesting as needed.
• Finds defects that manual code testing and reviews miss: It’s a known fact that manual code testing and reviews, even on projects demanding high code coverage levels, will miss important defects. Complex security vulnerabilities and concurrency problems are often missed with normal testing processes. Automating SAST throughout the SDLC will find more defects which will result in higher quality and more secure code as they are fixed.
• Prevents defects in the first place: Enforcing strict coding standards, such as MISRA C for safety critical software or SEI CERT C/++ for other applications, can help prevent many classes of defects in code. Enforcing good discipline in coding and creating a develop-analyze-test micro cycle for small code changes can prevent many defects from being created in the first place.  

Using a simple defect removal calculator, such as that proposed by Google, can illustrate a strong ROI for SAST solutions. Even though this is a limited view since it doesn’t account for significant downstream costs for major security vulnerabilities (more to be discussed on this coming up). However, even within the design, implementation, test and deploy cycle there is opportunity for improvements to security and reduced rework and retesting.

The following table is based on Google’s data and the type of applications they build. It does show that 40% of their engineering time is spent on bug fixing and on a large application that amounts to $2.4 million/year.

Table 1: Metrics from How Google Tests Software

Table 2: The cost to fix bugs at Google

So, what is the return on investment given these factors? SAST decreases the volume of defects in software under development at all stages of the SDLC. A simple analysis is to reduce the number of defects from the data we have from Table 1. Given this reduction in created defects during development, we can see a significant reduction in cost:

 

Defect Reduction from SAST	Estimated Savings
25%	$600,000
15%	$360,000
10%	$240,000

Table 3: The potential saving by reducing defects and vulnerabilities by 25, 15 and 10%.

This simple analysis yields significant savings when average defects are reduced. Even using relatively conservative figures for a single project, there are savings in terms of hundreds of thousands of dollars. This is based solely on reduced development and testing time and doesn’t account for costs to fix defects and vulnerabilities in production and deployment to customers.

SAST is More Than Just Defect Reduction

In addition to defect-detection, SAST solutions like CodeSonar from GrammaTech, are used to detect complex concurrency issues, and find errors that traditional testing methods miss. These critical benefits are not factored into the rather simple analysis above, but clearly add to the ROI of using a SAST solutions. However, finding defects that “slip through the cracks” give the greatest economic benefits to the development team.  

The Complex Answer: Beyond the Cost per Defect

As pointed out by Capers Jones (2012), looking at a cost-per-defect metric alone is misleading since it doesn’t factor in the volume of defects and the fact that the cost to find and repair a defect is often the same over time (something developers are quick to point out). Too many of these high-cost defects can seriously impact schedules and budgets. These costs often don’t include major cost overruns or cancelled projects. Cost per defect is important but so is the amount of them. Using the Google example above, how much does the impact of defects per thousand lines of code change as the product scales? Do the numbers hold true for a one million line of code project? In addition, how does the picture change when adopting DevSecOps and applying new testing solutions? Does an increased concentration on security from top down and across the SDLC change the picture?

Rather than just looking at the traditional cost-per-defect over time or per phase, which Jones argues is true mathematically but doesn’t reflect what is seen in practice, the more revealing data is the overall ROI from pursuing higher quality. In his research, the ROI of software quality is significant but relies on the maturity of the software development organization. Looking at Table 3, companies working on large projects with mature development processes that focus on quality and security are paradoxically removing less bugs than less mature companies – they are likely not introducing them in the first place. It’s clear that defect removal alone isn’t where they are saving money.

 

Application Size	10,000 Function Points, 1.25 MLOC
	Average Quality	High Quality	Difference
Defects Removed	50,400	33,600	-16,800
Defects Delivered	9,800	1,400	8200
Total Development Cost	$26,540,478	$18,361,525	$8,178,953
Maintenance Costs 1st Year	$8,000,000	$1,166,667	$6,833,333
Savings from High Quality	$15,012,287

Table 3: Savings from a mature, quality and security focused development approach versus average organizations on a large project.

Based on Jones’ research, it’s clear that the ROI in security and quality is tied to maturing the development process, like DevSecOps, to reduce the number of delivered defects. These companies not only reduce their development costs, they reduce downstream maintenance costs as well.

For more specific ROI numbers for shift-left test automation, where SAST plays and important role, Forrester Research found an ROI 205% over three years, with a real dollar return of almost $7 million on a $3.3 million investment. These benefits from shift left automation included increased output per developer, decreased testing time, improved risk avoidance and bug remediation. In the Forrester Study, the tools studied removed about 20% of the bugs from the software which aligns with the direct cost avoidance savings noted in Table 3.

The Exponential Cost of Failure

As seen above, best practices such as DevSecOps and automating SAST throughout the SDLC can produce significant savings by finding and fixing defects and vulnerabilities. The results are higher quality and more secure code that forms the foundation of the software applications or software powering devices. What you will see in this section is that there are other cost factors to consider when measuring the ROI of using SAST solution that are not as concrete to calculate.

The average enterprise individual data breach costs a company $4.24 million, the highest average total cost in the 17-year history of IBM’s annual “Cost of a Data Breach Report” for 2021. While this seems astronomically high, you have to consider all the factors involved in solving such a breach. Not to mention the lasting damage to an organization’s brand and reputation. In 2020, it was estimated that software defects of all kinds, including software vulnerabilities, cost the economy $2 trillion. Unsurprisingly, this is due to software defects making their way through the entire software development phase to manifest in products delivered to customers.

Here are some things to consider when evaluating the real cost of security vulnerabilities and other software failures:

Risk and liability are high with safety critical devices such as critical infrastructure controls, medical and automotive systems, and aircraft electronics. Failure here could cause human injury or even death. The Prius brake issue turned out to be a software failure that cost Toyota $5 billion to remedy which included the recall of four million vehicles. The Boeing 737MAX accidents and grounding of the airplane is likely to cost Boeing $19 billion.

Brand and reputation might be difficult to monetize but it certainly is a large problem for corporations that have fallen afoul of a large security incident. The Equifax breach and the more recent Solar Winds supply chain attack are two prominent examples. Data breaches increased by 17% in 2021 with several high profile cases like zero day vulnerabilities in Microsoft Exchange Server and the Log4J/Log4shell vulnerability.

Customer experience is a leading differentiator in many of today’s applications. Poor implementation (design and coding defects), poor security and poor quality all result in poor customer experience. For example, performance can be a significant customer experience issue: Amazon found that for every 100ms of latency in their online applications costs them 1% in sales. Google found that 500ms delay in search page results dropped traffic by 20%. Customers are flush with choices in today’s market, customer experience is key in keeping them.

Patching and recalls are inevitable when serious security vulnerabilities or defects are found. In the Toyota case, they had to recall four million vehicles to patch their software. It’s expensive to recall and patch your own software but you are also downloading huge costs on to your customers. Organizations are spending thousands of hours and millions of dollars on patch management for software deployed in their environments. There is both an internal and external cost for security vulnerabilities and defects in software. Every unpatched piece of your code in customer’s hands is a liability for them and you as it opens up new threat vectors.

Compliance, especially for public companies is a critical part of their business. Failure to manage the risk to your business due to security incidents can lead to heavy consequences from the Securities Exchange Commission (SEC) or Federal Trade Commission (FTC). For example, the Equifax data breach resulted in $575 million fines payable to the FTC and CFPB. The Home Depot breach was $200 million in fines and the Capital One breach resulted in $190 million. Note these fines are over and above every other cost and liability resulting from the breach.

Cybersecurity insurance coverage and premiums are beginning to be impacted by software quality, safety and security issues. Insurers will begin raising rates or possibly even denying coverage to organization not following DevSecOps best practices. The SolarWinds attack cost cyber insurance vendors more than $90 million.

There is tremendous opportunity in reducing these downstream costs with improved software development, shifting security left and automating testing practices.

Summary

Software failures and security vulnerabilities can have catastrophic effects in human and economic terms. SAST solutions are part of mature DevSecOps process that ensures continuous improvement and the development of secure and high-quality software. The ROI for SAST solutions is compelling, underscoring its critical role during development but also the results in downstream quality and security.

To learn more about GrammaTech’s static application security testing solution, CodeSonar, and how to achieve your ROI, please contact us.