ABSTRACT: Scientific advancement is fueled by solid fundamental research, followed by replication, meta-analysis, and theory building. To support such advancement, researchers and government agencies have been working towards a “science of security”. As in other sciences, security science requires high-quality fundamental research addressing important problems and reporting approaches that capture the information necessary for replication, meta-analysis, and theory building. The goal of this paper is to aid security researchers in establishing a baseline of the state of scientific reporting in security through an analysis of indicators of scientific research as reported in top security conferences, specifically the 2015 ACM CCS and 2016 IEEE S&P proceedings. To conduct this analysis, we employed a series of rubrics to analyze the completeness of information reported in papers relative to the type of evaluation used (e.g. empirical study, proof, discussion). Our findings indicated some important information is often missing from papers, including explicit documentation of research objectives and the threats to validity. Our findings show a relatively small number of replications reported in the literature. We hope that this initial analysis will serve as a baseline against which we can measure the advancement of the science of security.

Ehab Al-Shaer is a Professor and the Director of Cyber Defense and Network Assurability at University of North Carolina at Charlotte. His research interest include network security, security management, fault diagnosis, and network assurability. He can be reached ealshaer@uncc.edu.

Mohammed Noraden Alsaleh received his Bachelor of Science is a Ph.D. student in Computing and Information Systems at the University of North Carolina at Charlotte.  His research focuses on verification and synthesis of resilient network configuration. He can be reached at malsaleh@uncc.edu.

Mahran Al-Zyoud is a third year PhD student at the University of Alabama. His research interest include security and privacy. He can be reached at mmalzyoud@crimson.ua.edu.

Morgam Burcham earned her Master's Degreee from the University of Alabama in 2016. She is currently employd at SAIC as a Software Applications Tester.

Jeffrey C. Carver received the PhD degree in computer science from the University of Maryland in 2003. He is an associate professor in the Department of Computer Science, University of Alabama. His main research interests include empirical software engineering, peer code review, human factors in software engineering, software quality, software engineering for science, and software process improvement. He is a Senior Member of the IEEE Computer Society and the ACM. Contact him at carver@cs.ua.edu.

Hongying Du is Software Engineer at Cisco. She earned her PhD from the University of South Carolina in 2014.

Fida Gillani is a PhD student at the University of North Carolina at Charlotte. His research interests include network security and resilience. He can be reached at sgillan4@uncc.edu.

Jun Jiang earned his PhD from the University of North Carolina at Chapel Hill in 2016. He is employed by Two Sigma Investments. He can be reached atjingcj@cs.unc.edu.

Özgür Kafalı is a Postdoctoral researcher in Computer Science at NC State University. His research interests include artificial intelligence, security and privacy in sociotechnical systems, and computational logic. Kafalı received a PhD in Computer Engineering from Bogazici University, Turkey. Contact him at rkafali@ncsu.edu.

Akond Rahman is a third year PhD student at North Carolina State University. His research interest lies in the area of DevOps and mining software repositories. He is a student member at ACM. He can be reached at aarahman@ncsu.edu.

Laurie Williams is a professor and the interim department head in North Carolina State University (NCSU)’s Department of Computer Science. She is the CoPI of the NCSU Science of Security Lablet.  Her primary research interests are software security and software engineering. She can be reached williams@csc.ncsu.edu.