Winning Paper | Honorable Mentions | Award Ceremony | Review Team

The seventh NSA Competition for Best Scientific Cybersecurity Paper recognizes the best scientific cybersecurity paper published in 2018. Papers were nominated between January 1, 2018 through December 31, 2018 and 34 nominations were received. Three papers were selected for recognition.

Winning Paper

The winning paper is Evaluating Fuzz Testing by George Klees, Andrew Ruef, Benji Cooper, Shiyi Wei, and Michael Hicks. This paper was presented at ACM SIGSAC Conference on Computer and Communications Security (CCS '18) in Toronto. 

The research team investigated the evaluation process of fuzz testing tools. Fuzz testing tools help evaluate quality of software code by inputting large amounts of random data and monitoring the program for unexpected behavior. The researchers asked an important question, how to determine which tool is best. They began with a survey of the experimental evaluation contained within 32 fuzz testing papers. Their analysis identified problems in each of these evaluations so the research team developed their own extensive evaluation process and then experimentally characterized the configuration options. The standardized evaluations showed that existing ad-hoc evaluation methodologies can lead to wrong or misleading assessments. From this research, the team developed guidance on evaluating tools. 

The paper was selected because it embodies the attributes of outstanding science and the criteria of the competition: rigorous research, generalizable results and clarity of presentation. This paper is a step forward in bringing scientific understanding to the security community. It is grounded to current understanding by its methodological survey of evaluation practices, then advances the science through quantitative analysis and proposes conclusions that apply broadly in the fuzzing community. This paper is already having tremendous impact on fuzzing research, setting the standard for how evaluations should be done. It is bringing scientific principles, such as rigorous conclusions and reproducibility to an area in need of scientific understanding.

George Klees	Michael Hicks	Shiyi Wei	Andrew Ruef

George Klees is a computer science student at the University of Maryland currently enrolled in the Advanced Cybersecurity Experience for Stuents (ACES) program. This past summer, he was a software engineer for YugaByte of Sunnyvale, California to help them develop their PostgreSQL-compatible distributed database software. George has a depth of experience in software development, computer systems, reverse engineering, and cybersecurity red-teaming through his work experience and numerous personal projects.

Michael W. Hicks is a Professor in the Computer Science department at the University of Maryland, and is the Past Chair of ACM SIGPLAN, the Special Interest Group in Programming Languages. His research focuses on using programming languages techniques to improve the security, reliability, and availability of software. He has explored the design of new programming languages, formal methods, analysis tools, and testing technologies for helping programmers find bugs and software vulnerabilities. He has studied the end-to-end challenges of secure software development through the security-oriented programming contest, "build-it, break-it, fix-it," that he co-invented offered to students, including those in his Coursera class on Software Security.  He has explored technologies to shorten patch application times by allowing software upgrades without downtime. He has extensively explored synergies between cryptography and programming languages, and is currently exploring the synergy between fuzz testing and property-based random testing. He edits the SIGPLAN blog, PL Perspectives, at http://blog.sigplan.org/

Shiyi Wei is an Assistant Professor in the Computer Science Department at The University of Texas at Dallas. He obtained his Ph.D. in Computer Science from Virginia Tech in 2015, and then completed a postdoc at University of Maryland, Collage Park before joining UTD. The goal of his research is to make software analysis and testing practical for improving the security and reliability of real-world software. His research interests span the areas of Programming Languages, Software Engineering and Security. In the past, he has developed new bug detection and verification tools for software written in C/C++, Java, JavaScript and Android, using a wide range of techniques including dataflow analysis, abstract interpretation, and fuzz testing. His work has been published in top venues in his areas of interests (e.g., PLDI, FSE, and CCS).

Andrew Ruef has a PhD in Computer Science from the University of Maryland, where he was advised by Mike Hicks. His research studied tools and experiments for software security, with projects that measured secure software development, investigated fuzzing, and static analysis.

Honorable Mentions

The 7th Competition selected two papers for an honorable mention. The first paper, Continuous Formal Verification of Amazon s2n by Andrew Chudnov, Nathan Collins, et al, was presented at the International Conference on Computer Aided Verification, 2018. 

This paper discussed the researchers’ advancements in the application of formal methods. Formal methods are security techniques where mathematical approaches are used to verify that software is written correctly. These techniques are often applied to security critical code. The researchers reduced the manual effort required in revaluating code after modifications.  This makes it practical to continuously verify the functionality of code during the development and maintenance stages of the software lifecycle; greatly increasing the use of formal methods and their security guarantees. 

This paper was selected for an honorable mention because this paper’s impact on security assurance is very high as it addresses one of the enduring challenges of formal methods. The researchers make a foundational research advancement and also provide a solution. This paper describes the first integration of full formal specification and verification into a modern continuous integration development environment. The researchers enable formal methods approaches to be useful and practical in real life, a testament to the researchers’ breakthroughs. Their breakthrough work will enable developers in many fields to incorporate formal methods into their software development processes.

Andrey Chudnov	Stephen Magill	Eric Mullen

Dr. Andrey Chudnov received his PhD from Stevens Institute of Technology (Hoboken, NJ) where he did research on dynamic enforcement of information flow policies, especially by program transformation. After that he worked at Galois, Inc. (Portland, OR) where he contributed to the SAW program verifier and applied it to verifying correctness of Amazon s2n. He currently works as an Information Security Architect at the D. E. Shaw Group (New York, NY).

Dr. Stephen Magill is a Principal Scientist at Galois, Inc. and a world-recognized expert on programming languages and program analysis, with work ranging from development of high-level languages to static analysis of low-level systems code. He is also CEO of Muse Dev, a Galois spin-out focused on providing developers with low-friction access to static analysis results as an integral part of the development process. Stephen has a Ph.D. in Computer Science from Carnegie Mellon University, and his work has been widely published. He has led several research and development projects, including serving as principal investigator on a number of DARPA programs. Prior to Galois, Stephen was a research scientist at the Institute for Defense Analyses Center for Computing Sciences and a researcher at the University of Maryland. Stephen also serves on the University of Tulsa Industry Advisory Board and numerous program committees and funding panels.

Dr. Eric Mullen has a B.S in Computer Science from Harvey Mudd College, and received his PhD in Computer Science from the University of Washington in 2018. He currently works for Google, on the Asylo project. His expertise is mainly in verified systems, built in Coq.

The second paper to receive an honorable mention is Meltdown: Reading Kernel Memory from User Space by Moritz Lipp, Michael Schwarz, et al. This paper was presented at the 2018 USENIX Security Symposium. The paper detailed an attack to exploit a side effect of a computer processors’ out-of-order execution performance feature. The vulnerability allowed an attacker’s program to read almost any memory on a computer system including sensitive information such as password and cryptographic keys. Meltdown received worldwide attention when the information became public in 2018. 

This paper was selected because it is an excellent example of researchers challenging and disproving assumptions of the security community. The Meltdown exploit proved that the community had oversimplified the model of a processor. Instead of reasoning at the architecture level, the community had to look at the micro-architecture level. This paper publically opened up this new area of security research. The paper competition also recognizes the work the authors did to responsibly disclose this far reaching vulnerability.

Moritz Lipp	Michael Schwarz	Anders Fogh	Daniel Genkin	Werner Haas

Moritz Lipp is a PhD candidate in the CORESEC group at the Institute of Applied Information Processing and Communications at Graz University of Technology. He finished his master degree with distinction on the topic of cache attacks on mobile devices. Since then, his research focuses on microarchitectural side-channel attacks and has been presented on international conferences like Black Hat or CCC. He was a part of the team that uncovered the Meltdown, Spectre and ZombieLoad vulnerabilities.

Michael Schwarz is an Infosec PhD candidate at Graz University of Technology with a focus on microarchitectural side-channel attacks and system security. He holds two master's degrees, one in computer science and one in software development with a strong focus on security. He frequently participates in CTFs and has also been a finalist in the European Cyber Security Challenge. He was a speaker at Black Hat Europe 2016, Black Hat Asia 2017, 2018 & 2019, and Black Hat US 2018, where he presented his research on microarchitectural side-channel attacks. He authored and co-authored several papers published at international academic conferences and journals, including USENIX Security 2016, 2018 & 2019, NDSS 2017, 2018 & 2019, CCS 2019, and IEEE S&P 2018 & 2019. He was part of one of the research teams that found the Meltdown and Spectre vulnerabilities as well as the ZombieLoad vulnerability.

Anders Fogh is a Senior Principal Engineer at Intel. He has been involved in software development and information security for more than two decades and is an expert in reverse engineering.

Prof. Daniel Genkin is currently an Assistant Professor at the Department of Electrical Engineering and Computer Science at the University Of Michigan. Before that, he was a Postdoctoral Fellow at the University of Pennsylvania and the University of Maryland, where he was hosted by Prof. Nadia Heninger and Prof. Jonathan Katz. Previously he has been a Ph.D student at the Computer Science Department in the Technion — Israel's Institute of Technology where he was co-advised by Prof. Yuval Ishai and Prof. Eran Tromer.

Werner Haas is the Chief Technology Officer of Cyberus Technology, devising leading solutions to secure our digital era. Since disclosing the Meltdown vulnerability to Intel, educating people on the basic principles of operation and mitigation options has become a regular occupation, be it at international conferences, workshops, or in dedicated training courses. 
His expertise stems from working 10+ years at Intel in the Germany Microprocessor Lab and the Systems Architecture Lab in Hillsboro, USA.

The memory subsystem has been a common theme of his research activities with innovative ideas finding their way up to Intel board level and memory protection keys (MPK) becoming an official feature of the x86 architecture. Between leaving Intel in 2014 and co-founding Cyberus Technology in 2017, he briefly worked as external consultant and specialist for hardware-level programming at FireEye, a US-American cyber-security company. 

Werner graduated in Electrical Engineering in 1997 at the University of Erlangen-Nuremberg. He began his professional career at its Institute for Computer Aided Circuit Design, working closely together with Lucent Technologies on next generation networking equipment.

Award Ceremony

On 10 October, the National Security Agency recognized the winners of the Science of Security 7th Annual Best Scientific Cybersecurity Paper Competition at a ceremony which included a presentation of the winning paper and a panel discussion.

Review Team

NSA Competition Leads:

The following individuals served as distinguished experts for the 7th annual competition:

About the Competition

The Best Scientific Cybersecurity Paper Competition is sponsored yearly by NSA's Research Directorate and reflects the Agency's desire to increase scientific rigor in the cybersecurity field. This competition was established to recognize current research that exemplifies the development of scientific rigor in cybersecurity research. SoS is a broad enterprise, involving both theoretical and empirical work across a diverse set of topics. While there can only be one best paper, no single paper can span the full breadth of SoS topics. Nevertheless, work in all facets of security science is both needed and encouraged.