GrammaTech Adds Real World Benchmarks to SWAMP

JANUARY 2019

ITHACA, N.Y.

Software development and quality managers that are looking to measure the benefit of static analysis now have a platform to do just that. GrammaTech, under contract for the Department of Homeland Security (DHS) Science and Technology Directorate (S&T), has created independent real-world benchmarks that are now available in the Software Assurance Marketplace (SWAMP).

Several different synthetic benchmarks exist that can be used to measure how well static analysis tools perform in detecting bugs. However, many of these have limitations with the code paths typically being too simple. GrammaTech, under contract for DHS S&T, has created BugInjector, a tool that can inject Common Weakness Enumeration (CWE) based bug patterns into existing code bases, thus delivering real-world benchmarks. The BugInjector tool is available directly from GrammaTech to inject bugs into private code bases for training purposes. Additionally, four different real-world code bases (nginx, grep, sqlite, lighttpd) have been injected with bugs and are available through the Software Assurance Marketplace, enabling users to easily benchmark how well their static analysis tools are able to find these bugs in realistic code paths.

“There is an urgent need for benchmarks, such as those from GrammaTech, to allow software developers to evaluate static analysis tools in a comprehensive and real-world setting,” says Barton Miller, Professor of Computer Sciences at the University of Wisconsin – Madison and Chief Scientist of SWAMP. “Also, developers of static analysis tools now have the ability to enhance their tools or benchmark new static analysis technologies with realistic test cases. Integrating these benchmarks into the SWAMP platform increases their effectiveness and availability.”

“GrammaTech CodeSonar® has always focused on highest recall,” says Paul Anderson, VP of Engineering at GrammaTech, Inc. “Many tools claim that they can catch a particular CWE, but there has never been a way to measure how well tools perform if this CWE is hidden deep inside a code path. BugInjector provides an automated way to objectively measure static analysis tool recall; interested parties can now evaluate CodeSonar®’s market leading recall against other tools easily.”

The Software Assurance Marketplace’s static analysis capabilities are available for use in the cloud or on-premise at no cost. Interested parties can sign up to use the SWAMP at mir-swamp.org and find the BugInjector test cases on the Resources page under Packages. After selecting a package and version containing a CWE of interest, users can run an assessment of the chosen “bug injected” software using one or more software assurance tools. GrammaTech CodeSonar® is one of the commercial tools that is integrated into the SWAMP, along with many other open source static analysis tools. Users can also download BugInjector test cases to run against tools they are developing.

About GrammaTech:

GrammaTech’s advanced static analysis tools are used by software developers worldwide, spanning a myriad of embedded software industries including avionics, government, medical, military, industrial control, and other applications where reliability and security are paramount. Originally developed within Cornell University, GrammaTech is now a leading research center for software security and a commercial vendor of software-assurance tools and advanced cyber-security solutions. With both static and dynamic analysis tools that analyze source code as well as binary executables, GrammaTech continues to advance the science of superior software analysis, providing technology for developers to produce safer software. For more information, visit www.grammatech.com or follow us on LinkedIn.

CodeSonar® is a registered trademark of GrammaTech.

Want to see it in action?

Check out our demonstration video here.