A recent blog post, “Automotive software defects”, from Phil Koopman, Carnegie Mellon professor and author of “Better Embedded Software”, talks about increasing number of software defects in automotive software that are significant safety hazards. The post points out an increase in potentially life-threatening software defects being reported yet there is a general resistance in the industry to dealing with the quality, safety and security of the software.
As Koopman points out, “…at least some of these recalls sure look like mistakes that simply should not be happening in life critical software.” The post contains a list of recent NHTSA safety recalls related to software defects. Koopman is clear that isn’t to shame any particular manufacturer but rather “…the point is that safety critical software defects are both pervasive and persistent across the automotive industry… does not instill confidence about life critical software in a self-certifying industry that in the US is not required to follow international software safety standards.”
Notable Examples
Koopman’s blog contains a fairly long list of identified issues that have become NHTSA recalls, signifying that the problem is a significant enough safety hazard to warrant a recall. Here’s some examples that are directly related to faulty software:
- ABS and dynamic stability control (DSC) are disabled due to a fault in the diagnostic check at start up, disabling both systems. (NHTSA Recall 21V-167)
- Radio software security vulnerabilities that can be exploited to give unauthorized remote control of certain vehicle systems, increasing the risk of a crash. (NHTSA Recall 15V-461, 15V-508)
- Electronic stability program (ESP) makes vehicle pull to one side unexpectedly when performing an evasive maneuver. (NTSA Recall 21V-071)
- Integrated electronic brake system may detect an abnormal sensor signal decreasing braking performance. This defect was identified as a lack of proper fail-safe logic in the control software. (NHTSA Recall 20V-748)
- Electronic stability program issue that caused incorrect intervention due to defects in the yaw sensor software, particularly with the diagnostic procedures. Sensor drift in this case can increase the risk of a crash.
- Automatic emergency braking may not engage due to missing software code and incompatibilities with new hardware introduced.
There are plenty of other examples and it’s just a representative list. There are many more recalls like this, related to software and hardware defects, that effect safety critical aspects of stability, braking and engine control.
What Can Be Done Differently and How Static Code Analysis Helps
Obviously, the safety and security of automotive software is a systemic problem indicating the need for corporate culture, process and technology changes. There also issues with the industry being self-certifying and there isn’t universal adoption of standards like ISO 26262, at least with US manufacturers. Since that’s too large a topic to handle in one post, let’s focus in on some of the processes and procedures that can be improved with static application security testing (SAST) and software composition analysis (SCA.)
The Role of SAST in Safety Critical Software Development
The power of SAST is that it doesn’t rely on test cases to find problems, nor does it require that an error or failure be reproduced. An advanced SAST tool, like GrammaTech CodeSonar, can infer the runtime behavior of a program without actually running the program. Furthermore, when it identifies a problem, it also pinpoints the locations within the code that created the failure. This makes the job of debugging far simpler.
Static analysis does not eliminate the need for testing altogether, but it complements testing activities, providing an additional critical validation technique. The reality is that in large and complex software systems, there are so many possible states, and such an astronomical number of possible paths of execution, that it is infeasible to exhaustively test them all. Static code analysis, on the other hand, can explore these paths and state conditions in the aggregate, and is able to find problems that are missed by testing.
- Prevention through coding standards: Coding standards are an important part of safety critical software development because they define a safer subset of a programming language. Most used standard in automotive software is MISRA C and C++ (with honorable mention to the related AUTOSARC++ standard.) These standards define a subset of C or C++ which is better suited to safety critical software. The emphasis is on avoiding known dangerous language constructs and parts of each language with potentially undefined behavior. Enforcing a stricter safe and secure coding standard prevents many of the software defects in the first place.
- Code coverage isn’t everything: Many safety standards require high levels of code coverage (proof that tests executed most, if not all, statements and conditions). Although this is very exhaustive, it’s very expensive to do and must be repeated in each major phase of development (unit, integration and system testing). The criticality of the software dictates the level of coverage with some less critical software requiring no formal test coverage (e.g. aircraft on-board entertainment). Testing code coverage is one metric to evaluate software quality by, but there are cases where it doesn’t catch everything.
- Vulnerabilities and bugs that coverage-based testing miss: Testing software based on coverage metrics is inherently unit-based (although coverage is evaluated at a system level as well). Concurrency errors and security vulnerabilities are two key instances of defects that can be missed even during rigorous testing. Concurrency is often tough to program correctly and can yield errors (e.g. race conditions) that are undetected until some unforeseen condition during operation. Security vulnerabilities do manifest as bugs in the code — the conditions creating the error are often due to types of input not considered during testing. SAST can discover these errors early and prevent them from being show-stoppers late in the development cycle.
- Detect defects early: Rigorous testing can discover most defects in software, but it’s expensive and extremely time-consuming. Discovering and fixing these bugs when writing the code is also considerably cheaper than later in the development cycle (defect discovery is exponentially more expensive over time). Static analysis can detect bugs in the code as it is written — as part of a developer’s development environment — greatly reducing the downstream cost of defects.
- Analyzing open-source and third-party code: Use of third-party code and open source software is a fact of life in embedded software development. Some safety standards consider any software that isn’t developed to the specific standard as software of unknown pedigree (SOUP) — software that needs to be looked at carefully for inclusion in the system. Software composition analysis tools, like GrammaTech CodeSentry, can analyze third-party binaries to discover defects and security vulnerabilities and generate a software bill of materials (SBOM) – an increasingly important development artifact.
Integrating static analysis into your existing development process
Integrating SAST into any SDLC process is simple with the right tools and can be done at any stage. Furthermore, CodeSonar and CodeSentry do not require any changes to the build process, build tools, or source code. Once installed and configured, it can run seamlessly any time code is compiled. This makes it possible to find bugs as they are introduced, when the cost of fixing them is least.
Summary
NHTSA safety recalls for software-related defects is on the rise and it indicates that automotive software development still needs improvement. However, developing embedded software for automotive systems, requires a rigorous approach and commitment to increasing safety and security. Changes need to be made top down with cultural and process changes. It also requires bottom-up changes with best practices, including process and coding standards and other proven methods for improving code quality.
Advanced static analysis tools like CodeSonar, help automate enforcement of coding standards, and support standards compliance activities. More importantly, they play an essential role in finding defects in code that are missed during other verification and validation activities, and they aid developers in understanding and correcting code problems. Software composition analysis tools like CodeSentry can vet open-source and third-party software for known vulnerabilities and create software bill of materials (SBOM) to reduce the software supply chain risk.
Interested in learning more about CodeSonar? Schedule your demo today.
{{cta(‘25232961-7e2a-40ac-83b1-b2f534bbd51d’)}}
