We suggest that these problems can be avoided, thus making reported results more robust, by following some simple guidelines. Such guidelines are an instance of those developed subsequently by ACM SIGPLAN in an effort to improve the quality of empirical evaluations of automated testers, compilers, and analysis tools.

Michael W. Hicks is a Professor in the Computer Science department and the CTO of Correct Computation, Inc. He recently completed a three-year term as Chair of ACM SIGPLAN, the Special Interest Group in Programming Languages, and was the first Director of the University of Maryland's Cybersecurity Center (MC2). His research focuses on using programming languages and analyses to improve the security, reliability, and availability of software. He has explored the design of new programming languages and analysis tools for helping programmers find bugs and software vulnerabilities, and explored technologies to shorten patch application times by allowing software upgrades without downtime. Recently he has been looking at synergies between cryptography and programming languages, as well techniques involving random testing and probabilistic reasoning. He also led the development of a new security-oriented programming contest, "build-it, break-it, fix-it," which has been offered to the public and to students of his Coursera class on Software Security. He edits the SIGPLAN blog, PL Perspectives (https://blog.sigplan.org), and maintains his own blog, the PL Enthusiast, at http://www.pl-enthusiast.net/.