SoS Musings #63 - Testbeds for Security Experimentation

SoS Musings #63 -

Testbeds for Security Experimentation

Testbeds, as defined by Paul J. Fortier and Howard E. Michel, authors of "Computer Systems Performance Evaluation and Prediction," are system abstractions used to study system components and interactions in order to gain a better understanding of the real system. A testbed typically provides a realistic hardware-software environment for testing components in the absence of the complete system. It allows for a better understanding of the system's functional requirements and operational behavior. The testbed provides a laboratory environment in which modeled real-world system components can be experimented with, studied, and evaluated from various perspectives. Testbeds can be an important mechanism for evaluating security performance experimentally in multiple areas of research in the Science of Security (SoS) community. There are several examples of testbeds being made available to SoS community members to contribute to the research and development behind security advancements.

Researchers at the US Department of Energy's Pacific Northwest National Laboratory (PNNL) developed the CyberNET testbed to improve and enhance cybersecurity research. This testbed provides an isolated and dynamic testbed that can easily be configured and customized, allowing researchers to build, test, evaluate, or conduct - research in an enterprise-like environment. It makes use of cloud technology to support a customizable and controlled cyber environment in which realistic models can be run using real software. The CyberNET testbed is built on OpenStack software with scientific modifications, using Xen and Kernel-based Virtual Machine (KVM) hypervisors, and the Lincoln Labs cyber range toolkit from the Massachusetts Institute of Technology (MIT). Researchers can use the CyberNet testbed to generate cyber models, collect data for analysis, and document the performed experiment for results that can be repeated and reproduced. PNNL emphasizes that the testbed accelerates cyber research while lowering costs, time, and redundancy in the cybersecurity domain. Enhanced modeling and simulation, backed up by real-world data sets, increase model realism, resulting in more effective research.

Three new testbeds at Virginia Commonwealth University's College of Engineering, funded by the Commonwealth Cyber Initiative (CCI), aim to help researchers and business partners examine the security of medical devices, NextG applications, and smart city operations. The CCI is a collaboration of industry, higher education, and economic development partners that serves as a research, workforce development, and innovation engine at the intersection of cybersecurity, autonomous systems, and intelligence. The new testbeds include the NextG testbed, Medical Device Security testbed, and OpenCyberCity testbed. The Medical Device Security testbed is dedicated to testing medical devices to discover and develop suggestions for mitigating vulnerabilities, and the NextG testbed provides radio silence to enable 5G experiments in an isolated environment.The NextG testbed is where the networked underpinnings of many advanced applications in smart cities and medical devices are evaluated. Researchers are working on characterizing the emitted signals of compromised medical devices, which will aid in the development of detection systems to secure medical devices in networked health care environments. Furthermore, researchers in the NextG testbed are developing and characterizing new magnetic materials that may be useful for radio frequency shielding and power dissipation in telecommunications, security, medical, and smart city applications. The OpenCyberCity testbed runs experiments related to smart cities and autonomous vehicles on a realistic, small-scale cityscape. This smart city testbed consists of data collection and processing units, as well as database management, distributed performance management algorithms, and real-time data visualization. These testbeds are expected to add to the CCI network's capability to examine technology and applications from industry and government partners before they are widely used. Students will also be able to gain meaningful experiential learning opportunities by conducting research using the testbeds, thus helping them prepare for a career in the cybersecurity field.

A secure laboratory facility at the University of Maryland (UMD) is hosting a new 5G security testbed dedicated to commercial 5G networks. The Cellular Telecommunications and Internet Association (CTIA), which is a trade association representing the US wireless communications industry, created the testbed in collaboration with other organizations to test 5G security recommendations in real-world conditions using commercial-grade equipment and facilities. AT&T, Ericsson, T-Mobile, UScellular, MITRE, and UMD are its founding members, bringing industry expertise to the security testbed and bolstering its ability to improve the wireless security ecosystem and ensure strong safeguards on 5G networks. The 5G security testbed is built on both standalone and non-standalone 5G network architecture through cutting-edge equipment and facilities. In non-standalone architecture, the 5G network is built over an existing 4G network, while in standalone architecture, the network functions over its own 5G Core that can handle both 4G and 5G traffic. In the non-standalone architecture, an Ericsson Radio Access Network (RAN) at the UMD directs traffic from user equipment (such as 4G and 5G smartphones) to the Ericsson LTE Evolved Packet Core (EPC). Signaling and network user traffic are routed from the EPC to network databases, data and voice server gateways, and IP networks as needed. In the standalone 5G architecture, user equipment connects to the Ericsson Dual Mode 5G core at MITRE via the Ericsson 4G and 5G New Radios at UMD. The standalone architecture will enable a hybrid mode that supports VoLTE (4G) voice calls and pure 5G data at the same time, as well as an evolving suite of 5G security functions. In the future, this architecture will evolve to support VoNR (5G) voice calls. The 5G security testbed prioritizes use cases recommended by the Federal Communications Commission's (FCC) Communications Security Reliability and Interoperability Council (CSRIC) technical advisory body, covering issues such as 5G non-standalone (4G LTE core) and standalone (5G core) network security, new CSRIC recommendations for both 5G network environment, virtualized 5G network security, network slicing protections, and network roaming security. The outcomes of such use cases may inspire technology that will aid in the transformation of cities, governments, and industries, enabling applications such as private 5G networks for enterprises, dynamic supply-chain verification technologies, rapid threat detection, and more.

The National Institute of Standards and Technology (NIST) is promoting an experimentation testbed to address the changing cybersecurity landscape and rising threats to Machine Learning (ML) algorithms, providing researchers with a new way to test products against various attacks. The first iteration of the testbed, called Dioptra, was showcased at the National Cybersecurity Center of Excellence, a public-private sector collaborative hub within NIST's Information Technology lab. Harold Booth, a NIST computer scientist and project lead for the National Vulnerability Database, describes Dioptra as a testbed for evaluating which security techniques and solutions can be effective in protecting ML-enabled systems, thus allowing researchers to compare methods against a range of attacks under diverse conditions. The testbed is based on NIST Internal Report 8269, a taxonomy of adversarial ML published by the agency in 2019 that identifies three major categories of ML algorithm attacks: evasion, poisoning, and oracle. These attacks manipulate or alter test data in order to cause ML models to behave incorrectly, or reverse engineer models for the adversary's benefit. Researchers can use Dioptra to test multiple combinations of attacks, defenses, and model architectures within their systems to gain further insight into which attacks may pose the greatest threats, and what solutions may keep their ML algorithms the safest compared to other techniques. The modular design of the testbed allows researchers to easily swap in different datasets, models, attacks, and defenses, thus supporting the ability to advance the metrology required to help secure ML-enabled systems in the future. Dioptra is packaged with about ten built-in demonstrations of attacks and defenses from the literature that have been combined in various ways. The Fast Gradient Method evasion attack, Poison Frogs poisoning attack, and a membership inference Oracle attack are among the attacks. Among the defenses are feature squeezing, adversarial training, and jpeg compression. Contributing to the creation of more robust defenses against attacks on ML models, the Defense Advanced Research Projects Agency's (DARPA) Guaranteeing Artificial Intelligence (AI) Robustness against Deception (GARD) program researchers developed a virtual testbed called Armory. This testbed aims to enable repeatable, scalable, and robust evaluations of adversarial defenses as it provides researchers the ability to alter scenarios and make changes to ensure that defenses can deliver repeatable results across various attacks. Armory uses a Python library for ML security called Adversarial Robustness Toolbox (ART), which provides tools that allow developers and researchers to defend and evaluate their ML models and applications against different adversarial threats, such as evasion, poisoning, extraction, and inference.

Exploration of the cybersecurity domain in a controlled testbed environment is critical for researchers to understand risks, test defense, and develop prevention mechanisms. Researchers can identify solutions with predictable results by running controlled and repeatable cyber experiments in a testbed environment. It is essential to continue developing and supporting such testbeds for further SoS advancements.