Spotlight on Lablet Research #3 - Predicting the Difficulty of Compromise through How Attackers Discover Vulnerabilities
Spotlight on Lablet Research #3 -
Project: Predicting the Difficulty of Compromise through How Attackers Discover Vulnerabilities
Lablet: North Carolina State University
Participating Sub-Lablet: Rochester Institute of Technology
The goal of this project is to provide actionable feedback on the discoverability of a vulnerability. This feedback is useful for in-process software risk assessment, incident response, and the vulnerabilities equities process. The approach is to combine the attack surface metaphor and attacker behavior to estimate how attackers will approach discovering a vulnerability. The researchers want to develop metrics that are useful and improve the metric formulation based on qualitative and quantitative feedback.
Led by Principle Investigator (PI) Andy Meneely and Co-PI Laurie Williams, this project focuses on the attack surface based on the notion that pathways into the system enable attackers to discover vulnerabilities. This knowledge is important to software developers, architects, system administrators, and users. A literature review to classify attack surface definitions led to six clusters of definitions which differ significantly (methods, avenues, flows, features, barriers, and vulnerabilities). The methodology used to discover the attack surface (mining stacktraces from thousands of crash reports) and what the attack surface meant within the context of metric actionability, will lead to evolving the models for a risky walk and deploying a human-in-the-loop study. Attacker behavior data is gathered from the National Collegiate Penetration Testing Competition (CPTC) from years 2018 and 2019.
A national collegiate penetration testing competition allowed researchers to collect a massive data set. Nine teams from schools across the United States performed coordinated sets of network and device penetration attempts during the 2018 National Collegiate Penetration Testing Competition (CPTC). During the ten-hour attack window, the teams generated an aggregate of more than 300GB of alert logs across duplicate networks consisting of 252 virtual machines in total. Researchers captured and made available full images of 99 virtual machines (as .vmdk) and approximately 200GB of alert log data (as JSON) for the six teams who consented to allow their data to be published in this academic research. The inclusion of virtual machine instances along with network alert data is a novel contribution to the research community.
This national collegiate penetration testing competition data set enables the research team to provide a fine-grained history of vulnerability discovery and exploitation. With this data, they can enrich their models of the attack surface, which will, in turn, lead to more robust metrics of difficulty to compromise. Given that this data is from a competition where teams were assigned the same systems and evaluated on their attacks, the difficulty to compromise will come from the correlation of competition data with the alert and virtual machine data.
In completing the collection and analysis of the CPTC data set, project researchers manually annotated and curated the data from the CPTC data set and have over 400 events logged with 79 vulnerabilities reported. They classified each event as part of the MITRE ATT&CK framework, with reviewer agreement kappa scores over 80%, and have begun their attacker behavior model analysis. As part of their submission to the ESEM 2019 New Ideas and Emerging Results track (ESEM NIER), they honed their annotation process by curating one team's entire timeline. Then, to scale up the approach, they hired a CPTC-trained competitor to capture the timelines and map the events to MITRE ATT&CK. The resulting data set will be a fine-grained log of events that map the techniques attackers use to break into systems in the competition environment. The next step for this data set will be to create a probabilistic model that can estimate the probability of discovery for a given vulnerability based upon attacker behavior. A single team had 47 relevant events with approximately 17 vulnerabilities found, which was filtered from millions of events from the SPLUNK monitoring system.
The researchers also collected data from the 2019 CPTC competition, which included over 7TB of logs ranging from command histories, process information, network metadata, and a variety of other fine-grained measurements from the Splunk incident response system. Their techniques for gathering data and creating timelines were optimized ahead of this competition through research assistants volunteering to contribute to the competition’s monitoring team. Project researchers were able to create example infrastructures and tested out monitoring tools to ensure reliable data collection. With this data, they can re-create the timelines and deliver them faster to the research public.
Defects in Infrastructure as Code (IaC) scripts can have serious consequences for organizations that adopt DevOps. While developing IaC scripts, practitioners may inadvertently introduce security smells. Security smells are recurring coding patterns that are indicative of security weakness and can potentially lead to security breaches. The goal of this work is to help practitioners avoid insecure coding practices while developing IaC scripts through an empirical study of security smells in IaC scripts. They expanded the scale and depth of previously-reported security smell work by increasing the number of languages to three: Ansible, Chef, and Puppet (prior reported results were for the Puppet language only). They identify nine security smells for IaC scripts.
Security testers often perform a series of operations to exploit vulnerabilities in software source code. Project researchers are conducting a study to aid them in exploiting vulnerabilities in software by synthesizing attack mechanisms used in open-source software. By characterizing these operations, they will identify patterns of tools and techniques used to exploit a vulnerability. This information will help security testers detect vulnerabilities before code changes are integrated and deployed. They are currently investigating security reports archived in open source software bug reports, such as the Bugzilla reports, hosted by Mozilla to characterize what steps are executed to exploit a vulnerability. They are using the Structured Threat Information eXpression (STIX) format to record and standardize the steps needed to exploit a vulnerability.
The work done under this project will lead to the development of a new set of metrics for measuring exploitability using the attack surface. These metrics are based on the behavior observed by penetration testers in a competitive environment. The intrusion detection data collected from the CTPC have provided detailed timelines of how attackers find, exploit, and pivot with vulnerabilities. When studying how they work with the known attack surface, researchers will develop metrics that show which vulnerabilities are at the highest risk based on the current deployment.
Additional details on this project can be found here.