ABSTRACT

Organizations have little idea of how secure they are against varying degrees of attacks. Even after expensive penetration testing that may reveal vulnerabilities, an organization faces the difficult process of finding a security product that improves their security posture. We propose a novel scalable method for answering vital questions such as “How secure is an organization?” and “What is the most effective additional security product to purchase and deploy?” by empirical experiments. We acquired detailed attack information appropriate to represent a wide range of attacks seen in the wild. Each attack type is then tested against a large variety of security products each tested against the portion of the attack that it can parse. We then determine which attacks go undetected by a set of security products. This test determines the overlap of security products and which additional product would best fill the gaps in any particular security setup.

This approach also may be used to measure the overall security posture of an organization. We conduct an experiment implementing the approach for the subclass of attacks known as drive-by downloads and the subclass of adversaries that send such attacks in non-targeted spam emails. We measure over 40 security products varying in detection techniques. By capturing in the wild attack data and testing against these products we identify in real time the particular attack vectors each is able to detect. These measurements reveal the redundant and complementary layers of defense against drive-by downloads confirming the need for defense in depth even against non-targeted attacks. The technique is clearly extensible and may be implemented as a cloud-based service to amortize its cost by testing a number of (DoD) organization's network flows that subscribe to the service. This work is an initial practical step towards a Science of Security.