We need to know what’s in the software that is supporting our business.
One of the things I love about my job is that I get to talk to lots of people, across industries, about the challenges they are facing with software supply chains. What helps me a lot in these conversations is that I was once exactly where they are as part of an Open Source Governance committee. I was tasked with overcoming all of the barriers to using open source software at a risk-averse Fortune 100 company. Today, the barriers to use aren’t the issue anymore, it is dealing with the ever-shifting landscape of ‘known vulnerabilities’ and building the muscle to quickly respond to these types of events before being negatively impacted.
There is no better example than the recent Log4j remote code execution (RCE) vulnerability from late last year. Every conversation I’ve had this year has been in response to the pain felt by the whole community. It was all hands on deck, vacation or not, and I’ve heard some teams estimate 50-100 weeks of work being done in 8-10 days. That’s the kind of trauma that sticks with you. What struck me in particular was how this event spilled over into vendor relationships. For the first time, Third Party Risk Management (TPRM) teams were pulled in.
For the first time ever, I saw customers not only ask if vendors were vulnerable, but they asked “Which version of the patch did you apply?” “What day did you patch?” and other very specific questions that I had never had to field before. –Tanya Janca’s personal blog, Why Can’t I Get Over Log4j?
Reading that article was my own “a-ha” moment, especially when combined with the stories I was hearing on Zoom calls with organizations struggling to mitigate the risk. I had seen the value of software bills of materials (SBOMs) during my time as a DevOps practitioner, but back then I was focused on building quality into the software we produced, which you should all absolutely be doing. However, the prevalent use of open source software and libraries in software development often gets overlooked and introduces significant risk if not properly vetted and managed. If we, as software consumers, should all be managing our third-party dependencies, then why is it that none of your software suppliers can produce one for you?
I’ve filled out my share of security surveys over the years, and there are all sorts of questions in there trying to figure out if you’re running a real business and have various safeguards and practices in place. I’m willing to bet that TPRM teams could ask ten questions about software development and testing practices and not learn as much as just asking if they are willing to share an SBOM. I can distill all of the conversations I’m having into this common sentiment: We realized we had a gap in our ability to understand if this risk existed in our commercial off the shelf (COTS) software applications and, to make things worse, we had no leverage in our contracts to hold them accountable for vulnerabilities in open source components that put us at risk of cyberattacks.
The real problem, in my opinion, was an inability for software producers to communicate quickly and efficiently with their customers. I’m pretty sure the lion’s share of those 50-100 weeks’ worth of effort were spent on just figuring out where Log4j was being used in the hundreds to thousands of applications and software packages in the enterprise. Teams that had already adopted software composition analysis (SCA) practices into their development process likely had a head start, but then there is always more software out there than what is currently being developed. In the end, it all devolves down to using endpoint scanning to go re-discover what you should have already known. Available SBOMs show the use of the Log4j library could have reduced time spent on forensics and allowed cyber defenders to quickly apply security controls around the applications that were known to be impacted.
Where do we go from here?
First, I want everybody to go update their ‘Security Surveys’ to include two new line items. Ask your software vendors to provide an SBOM and a Vulnerability Data Report. Companies that can do these two things are demonstrating they are in control of their product and deeply understand how it operates. No number of other questions about third party attestations or pen testing will provide any more useful information than those two questions.
We all need to realize how important this level of transparency and detail is. If you are writing software, make sure you are capturing SBOMs as part of the software development life cycle (SDLC). If there are vulnerable components flagged in the SBOM, be prepared to provide the reasons these components do not impact the product.
Secondly, think about how you might negotiate your terms and conditions differently if the vendor cannot (or won’t) share an SBOM with you. Minimally, you’d want to secure our rights to do security research on the binaries for the purposes of identifying risk and interoperability with our security controls. The TPRM professionals that I have spoken with also want to incorporate SLA’s into the contract to avoid the lack of leverage they experienced when Log4j hit. I’d also consider asking for an additional discount on price when the vendor cannot provide an SBOM in recognition that the buyer now has to do the work necessary to ensure that the software product they are buying and deploying does not introduce hidden risk. Hopefully, this will motivate vendors to start producing SBOMs!
In the absence of SBOMs, we can turn to tools that are capable of scanning these products to produce a SBOM and in turn correlate those findings with known vulnerabilities (CVEs). Not every CVE presents risk, but at least now we can have a conversation with our vendors, with SLA’s in place, to determine the actual level of risk. I’ve seen many instances where the library is not in use and can be removed, and other times where it simply needs to be updated. Tools available that enable you to analyze software binaries to identify open source components, produce SBOMs and generate vulnerability reports include CodeSentry from GrammaTech. Producing SBOMs on the software you are consuming will allow you to take a “trust but verify” approach to working with your software vendors so you can ensure a stronger enterprise security and risk posture for your organization.
