The adoption of any new tool into an existing software development process and established code base is always a challenge. Static analysis tools are no different but there are steps to take to make the transition easier and smooth the introduction of these tools into an existing workflow. In addition, the techniques outlined here are useful on an ongoing basis to introduce new static analysis features and get the most return on investment from static analysis.
Getting the Most Out of Static Analysis
The key steps to dealing with legacy code (and for any new adoption of static analysis) are as follows:
- Determine the end goal: Without a clear goal it’s difficult to measure success against. Is the goal to increase security? Reduce the number of defects in delivered products? Achieve compliance with a coding standard? It’s critical to set goals as this drives the way static analysis tools are to be used within the development process. Focusing on a key criteria also gets the most value out of tool adoption.
- Establish a baseline report: After initial setup and installation of the static analysis tools you start by reviewing the first complete warnings report. This initial report is an important starting point for adopting static analysis into the project. The key thing here is to not be intimidated by the number of warnings. There are strategies to deal with these over time and not all warnings are of equal severity. In the case of GrammaTech CodeSonar, all warnings have a score, useful for identifying and fixing the most dangerous problems first.
- Stop the bleeding: In order to start improving the quality of a project, it’s important to make sure that no new defects are being added as code is changed or new code is added. This is done by creating a comparison between the latest report and the baseline report to look for new warnings that were introduced due to new code, or changes to old code. The priority should be on fixing any new defects introduced during each iteration.
- Tackle the backlog over time: Static analysis reports don’t add to existing technical debt but rather shine a light on what’s actually there already. At first, static analysis on existing and legacy code can often seem daunting because of the number of warnings, especially if the code base is large. Not all warnings are high severity and the goal does not have to be to reduce warnings to zero. There are other strategies that could make more sense, such as ‘fix all security bugs with a score over 50’. Using the filtering and searching tools built into advanced tools such as CodeSonar helps prioritize the warnings for remediation. Concentrate on the important defects to get the most ROI from static analysis.
Configuration, Searching and Filtering
An often overlooked aspect of static analysis tools is their data management capabilities. When dealing with large reports on multi-million lines of code, managing the data produced is key to making the tools useful. There are several ways to help developers focus on the most important warnings and this can be handled in several ways and often in combination. Most static analysis tools offer some kind of configuration options which determine which warning types are enabled/disabled. It’s also critical that each warning be stored with a state and history so it can be tracked over time. Warnings need to classified by severity and in the case with CodeSonar are given a score which combines both severity and likeliness of warning be a real defect, a “true positive.” Let’s consider these aspects in a bit more detail:
Configuration
Using CodeSonar as an example, it ships with a default set configurations that are a recommended baseline of important warnings that span the various class groups. The default presets offer an increasing depth and breadth of warning classes:
- Intro; great to use when first introducing static analysis to a project, it focusses on the most dangerous classes with the highest precision.
- Default; to dive a bit deeper, this preset provides deeper coverage for critical defects in each of the significant classes
- Thorough; increases the compute time allowed for the analysis engine and hence finds deeper, more complex warnings.
Of course, these presets can be customized further, for example to configure only the warning classes that are the most important for each software team. This is a good idea if certain warning classes are important to a project, for example, to make sure source code complies with MISRA C. There might be error classes that aren’t considered important and can be permanently removed from the analysis by configuration.
Searching
CodeSonar offers a flexible search mechanism (with the ability to save searches) that helps narrow down a large list of warnings to a much more manageable amount based on state, warning score, error class, significance and many other factors. Filtering can of course be done on the attributes that have been assessed, or on some of the other attributes associated with a warning.
Filtering
The filter and focus approach attempts to address high priority warnings as part of the current workflow rather than at a later date. This is great for isolating specific warnings, for example, search for buffer overruns in code that has an external interface. Another example is to filter out by component such as warnings in third party code, or test code.
Assessing Results
It is important to realize that any assessment done on warnings is persistent from analysis run to analysis run. If something is marked as false or a real defect, there is no need to redo these assessments in the future and this work is not lost. This is a tremendous productivity booster compared to more compiler-like warnings that many people may be familiar with. There are multiple different attributes that are stored with a warning and that are part of the assessment that a software engineer can do:
- The State of a warning indicates its place in the assessment process. As warnings are reviewed they are placed in different states based on the outcome of the analysis and remediation. For example, a warning that turns out to be a real error can be assigned to a developer to be fixed and marked as “assigned.”
- The Priority of a warning is typically used by the engineer to indicate the urgency of a warning, or to suppress it in most searches.
For a discussion on assessing results, true and false positives see this previous post. To summarize: The usefulness of a static analysis tool is whether it can find a reasonable number of important bugs in legacy code without overwhelming developers with warnings, all without using an unreasonable amount of computing resources. However, even a tool with perfect recall (the ability of a tool to find real defects) can be worse if it also has poor precision (tool’s ability to exclude false positives). Too many false positives can drown out the true positives, meanwhile, focusing solely on reducing false positives results in missed true defects. The success of static analysis tools relies on their ability to balance precision and handling of false positives.
Summary
The adoption of static analysis as part of a development project with a significant portion of legacy code may seem daunting at first. However, there are simple techniques that can be applied to reduce the initial volume of warnings to make the tools and the process more palatable to new users. Using the approach of deferring less critical issues and preventing new bugs from entering the code base delivers the best return on investment for static analysis. Over time, the team can explore the backlog of warnings and on a priority/severity basis.
