AppSec Shift Left
A Study on Enterprise Application Security
Recent software supply chain attacks have revealed that fallout from compromised applications is a growing problem — for the targeted businesses, their partners and customers alike.
Despite recent attention, security transformation has lagged behind digital transformation. As AppSec and engineering leaders look to address modern DevSecOps challenges, they are contending with a lack of application security solutions that fit their needs:
Static application security testing (SAST) tools are based on outdated technology that hasn't changed much in the last 15 years. Full custom code analysis is still very slow, discouraging its users from running frequent scans.
Vulnerabilities in open source libraries are common issues. While software composition analysis (SCA) tools are identifying known open source vulnerabilities effectively, the high volume of issues uncovered combined with labor-intensive remedies mean fixes are often time- and cost-prohibitive to implement. Lookup isn't enough.
In this report, ShiftLeft examines how Next-Generation SAST and Intelligent SCA increase the speed of vulnerability scans narrow the scope of work by highlighting reachable issues, leading to better outcomes: more frequent scans, fixes earlier in the CI/CD pipeline that prevent security debt from accruing, and more security fixes overall. The data for this report represents customer usage of ShiftLeft CORE between May 1, 2020 and April 20, 2021.
Speed and Frequency of ScansExpand
While legacy security analysis tools can take hours or even days to conduct a full scan, ShiftLeft customers experienced a median scan time of 2 minutes and 20 seconds.
With shorter scan times, 46% of applications are scanned at least weekly and 17% are scanned at least daily.
Prioritizing Findings for Modern ApplicationsExpand
Legacy analysis tools generate a large number of false positives that can overwhelm AppSec and development teams.
When open source vulnerabilities are prioritized by accounting for true “reachability,” ShiftLeft found that organizations reduce the number of their SCA tickets by an average of 92%.
Fix-Rates for Managed CI/CDExpand
When increasing the speed and frequency of scans and prioritizing SCA tickets, ShiftLeft found enterprises that tightly integrate security testing within their CI/CD pipeline fix 91.4% of new issues.
Overall, customers fixed 58% of new issues before they became technical debt.
Fix-rates for managed CI/CD
A key concept behind “shifting left” is that an issue costs less to fix the earlier it is detected in the software development cycle. If vulnerabilities can be detected as soon as the application is “created,” the engineer who wrote the code can implement the fix while the functionality is still fresh in their mind.
Therefore, the true measure of shifting left is not the speed of the scan. That is just the initial and enabling step.
The most important outcome is that more issues are fixed before they enter production where they add to an application's risk and become debt for future sprints.Read more
of New Issues are Fixed by Enterprises with CI/CD Controls
Enterprise customers with hundreds of apps in development were able to fix 91.4% of new vulnerabilities within one to two sprints.1 This achievement was realized by customers who tightly integrated security scans with their CI/CD pipeline and performed multiple scans during each sprint.
These AppSec teams generally felt comfortable with security debt and were instead focused on preventing new vulnerabilities of certain types and severities from ever merging into their main branch. They integrate ShiftLeft with their pull request process and use build rules to manage risk in their software releases.
*For theses customers, the typical sprint is two-weeks long
More New Issues Fixed Than Debt Across All Customers
Across all customers, including enterprises and startups, the fix rate for new debt was 1.9 times more than for security debt.
The average fix rate for new vulnerabilities was 58% while the fix rate for security debt was 30.7%.
This leads us to conclude that the average customer is seeing great efficiency by “shifting left” and having the original developer remediate issues in their own code.
This argues that if customers using legacy code analysis solutions were using ShiftLeft instead, their technical debt would be halved. In other words, with one year of ShiftLeft use, they would reduce their application security risk by half.Read more
More Debt is Addressed by Startups than by Enterprises
We labeled another group of customers using CI/CD controls “born-in-the-cloud” startups. These customers have a very high fix rate overall.
While their new issue fix rate of 80% is lower than that of the enterprise customers who tightly integrate ShiftLeft into their CI/CD pipeline, their fix rate for security debt is significantly higher. At 69.9% it is almost 3.6x of that of the enterprise group.
The security teams at these startups are closely monitoring new releases but they are also evaluating all issues, including technical debt.
“As AppSec & DevOps partner together as coaches and DevSecOps enablers for engineering teams, developer-focused tools that are fast, accurate, and easy-to-integrate make it much easier to build trust and accelerate adoption.”
Jimmy Xu - Director of Application Security, Trace 3
Speed and Frequency of Scans
The ideal point for a security scan in the development process is at the pull request itself. This is the first time where the modified branch is in a complete state and the application can be tested as it will be deployed with all of its dependencies. However, this part of the process is sensitive to delays.
Development teams each have their own tolerance for testing delays but there is a common notion in DevOps of a thirty minute window where all build, unit, and other tests are expected to conclude for the majority of applications.
If a security scan falls comfortably in this window, Security and Engineering teams will integrate it into their build cycle. Large, monolithic applications are typically ineligible for this window based on build time alone. But when these applications can be scanned in under an hour, daily testing becomes possible.Read more
is the Median Scan Time Across Languages
Across all customers and all supported languages, the median scan time for ShiftLeft CORE was 2 minutes and 20 seconds.
This number represents the elapsed time from the creation of a pull request to the moment scan results are displayed in the CI/CD pipeline and in the ShiftLeft dashboard.
of Applications are Scanned at Least Weekly; 17% at Least Daily
Customers used ShiftLeft CORE to scan 46% of their applications at least weekly. They scanned 17% of applications at least daily.
These results are very high for a security scan. Published industry reports cite customers scanning less than 5% of applications weekly and less than 0.5% daily. ShiftLeft customer data is strong evidence of customer desire to scan more frequently.
is 95th percentile Scan Time for Compiled Languages
The 95th percentile scan time for compiled languages, Java and C#, was 17 minutes and 37 seconds and for scripting languages it was 11 minutes. This gives us a rough idea of how scan times for monolithic applications, which are typically compiled, compare to those for microservices which are typically written in scripting languages.
There is also clearly a correlation between compiled languages and more complex applications. Increased complexity and size of an application increases the number of expressions in the ShiftLeft code property graph.Read more
“Measuring and reducing false positives in security testing is important if you want to scale your program to meet the demands of modern dev teams. You need the visibility of static analysis but you also need to improve its results so the process is supported by devs and management.”
Paolo Del Mundo - Head of Application Security, The Motley Fool
Prioritizing Findings for Modern Applications
Just as a security test must fit into the DevOps workflow, the results from that test must fit into the AppSec workflow.
Tests generate vulnerability findings in custom code, open source, APIs, and other elements of a modern application. Each of these findings represents work for a developer that the AppSec team must first assess and prioritize.
Findings that are not legitimate are said to be “false positives.” Large numbers of false positives cost time to review, do nothing to improve the security of an application, and reduce developer confidence in the testing process.
For an AppSec team to efficiently analyze findings and maintain the momentum of DevOps, security tests must be accurate and provide context around security issues.
of All Discovered Issues Were Marked ‘Ignore’
In a typical sprint, customers would mark 2.5% of discovered issues as ‘ignore.’ While this does not mean the false positive rate is 2.5%, it is nonetheless an indicator of false positives. For the remaining vulnerabilities, which include both new issues and security debt, 45% were fixed within the sprint while the remaining 53% are assumed to have been in different stages of triage or progression.
There are several explanations for why an issue is neither fixed nor ignored. The issue could have been deprioritized and still has yet to be categorized. It could be that the issue has no easy fix but cannot be ignored. It is also possible the issue was mitigated outside of the application itself. This is a common choice for vulnerabilities like cross-site scripting that are “globally” mitigated.
What is clear, is that developers who deal with these issues and scan frequently need better triage tools. As engineers scan throughout the week, unfixed and unignored issues will keep showing up. This is an issue we intend to focus on in our next report.Read more
of SCA Tickets Reduced by Account for “Reachability”
In the timeframe of this report, Intelligent Software Composition Analysis (SCA) combined with static analysis was only available in the last two months. In the time it was available, it showed a 92% reduction in the number of tickets for known open source vulnerabilities. This was performed on 718 open source packages with a total of 2,761 known vulnerabilities. When ShiftLeft CORE analyzed these packages in the context of custom code, it was able to determine that only 200 of the known vulnerabilities were actually reachable through attacker controlled paths.
Substantial developer time is reclaimed by limiting tickets to those issues that actually pose risk to an application. Upgrades to open source issues consist of several steps and require scheduling that commonly pushes them out more than a month from discovery. But, just to consider the active developer time spent changing the code, assume a conservative estimate of 30 minutes per issue. This results in a 100 hours (200 issues x 30 minutes) spent addressing the open source risk cited above instead of 1,380.5 hours (2,781 issues x 30 minutes), a savings of 1,200 hours.Read more
AppSec scales during the COVID crisis - 2.4x increase in scans between Feb and March 2020
As a side note to this report, we saw a strong surge in frequency of scans in March of 2020. The average for scans in the months prior to March was around 11 scans per application per month. In March the average jumped to 26 scans per application per month and stayed there.
Many analysts have commented on the accelerated move to the cloud in 2020. We are encouraged to see that the applications in this report appear to have kept up with the need for increased agility arising from digital transformation.
Security Fixes by Type
The final element of AppSec productivity is the developer’s ability to remediate issues once they are discovered and verified by the AppSec team. The most proven strategy for remediation is a healthy working relationship between AppSec and Dev. A security testing tool can contribute, however, by providing Devs with context and education around the issue.
Scan results that are aware of the data flow through an application can provide this insight to the developer so they can quickly map the logic of the vulnerability in their code.
A tool can also provide contextual education that matches vulnerability type so devs have access to relevant assistance at the time they need it.Read more
of Fixes were for Critical or Well-known Issue Classes
When all of the fixed vulnerabilities are broken down by type they fall into 45 categories. 86% of total fixes fall in categories that are either well-known or considered critical. These categories include typical common web application vulnerabilities such as cross site scripting (XSS), SQL injection, directory traversals, insecure cookies and deserialization vulnerabilities.
10 Most Fixed
Issue Types Were All in the OWASP Top Ten
When issue types are ranked by fix rate, the ten most fixed vulnerabilities are related to injection or system takeover attacks. Again, here, we see a link between commonality and preference to fix. If developer familiarity is the key factor, an assumption could be made that a focus on diversifying developer education would raise the fix rates of other, less common vulnerabilities.
There also appears to be correlation between the complexity of the vulnerability and the percentage of fixes. This is not necessarily a case of engineers focusing on easy work. Certain issues might be better addressed in aggregate than by the sole efforts of individual developers.
For example, in Java, an insecure cookie can be fixed by adding a call to `setSecure` for the cookie in question, a fix that is simple, localized and conclusive. In contrast, a developer could try to fix a cross-site scripting issue locally, where the attacker controlled data is received or written into an HTML page, but it is typically not the best approach. The more conclusive fix is global and may require a redesign of the Web application. For example, the application might need to be redesigned such that all data written into HTML pages undergoes output encoding via a suitable library.
More broadly, in cases where a vulnerability class simply has not been considered at design phase but shows up in great number during a scan, developers may rightfully determine not to fix one vulnerability at a time but instead leave the vulnerabilities unfixed and perform a redesign instead.
“The most common vulnerability types have not changed much over the years yet we keep finding them in code. Developers need more thorough and focused education than what has been available on message boards.”
Malcolm McDonald - CEO & Author, Hacksplaining
A Word From ShiftLeft
As digital transformation continues to go mainstream, a security transformation must follow. Marc Andreessen’s claim that “software is eating the world” continues to prove valid as entrepreneurs still find new industries to disrupt with SaaS. But, with software comes security risks that AppSec and engineering leaders must manage.
Failure to do so is public and costly, and threatens to erode general faith in the digital transformation through headlines about compromised customer and employee privacy, theft of intellectual property, and outright extortion.
ShiftLeft was founded with the mission to help application security and development teams release secure code at scale. ShiftLeft CORE uses GraphDB based dataflow analysis to reveal the risk profile of a software build within minutes of a pull request. This process provides accurate findings without the need for machine learning models that may suffer from a limited training set. Additionally, CI/CD integration and build rules allow teams who have developed trust in the tool to customize and automate the risk that is allowed into their code base.
ShiftLeft CORE marks the first time AppSec teams have been able to get quality analysis of a full application so quickly. Legacy scan times are measured in hours or days and generate findings that need to be filtered by systems with human training. Until now, it has been an open question as to how teams would respond to early, frequent feedback. Would they embrace the information and use it to create more secure code? Or would they shy away from a system that could add more fix requests to their sprints?
The process behind this achievement can be seen in hard numbers: 46% of applications are scanned at least weekly.
The success of “shifting left” is exemplified by customers who fix 91.4% of new issues before they become security debt using ShiftLeft CORE. This achievement is the result of hard numbers: 46% of customers scanning weekly or daily, those customers seeing a 92% reduction in the number of open source tickets, and developers fixing new vulnerabilities in the same or following sprint in which they appear at 1.9 times the rate they address debt.
The idea that a company with this technology and these processes could have half the security debt that it does today means that software can become more secure even as development speeds up. But, we still have more to do.
Just as we see promise for this new process, we also see room for improvement. In particular, developers addressing risk at this frequency need better tools for triage.
This is just the beginning, so we hope you follow us on this journey to build the foundation for a more secure internet by helping developers build inherently secure applications.