Discovering Emergent Behavior From Network Packet Data: Lessons from the Angle Project

Abstract

We describe the design of a system called Angle that detects emergent and anomalous behavior in distributed IP packet data. Currently, Angle sensors are collecting IP packet data at four locations, removing identifying information, and building IP-based profiles in temporal windows. These profiles are then clustered to provide high-level summary information across time and across different locations. We associate certain changes in these cluster models with emergent behavior. Emergent clusters identified in this way are then used to score the collected data in near real time. The system has a visual analytics interface that allows different emergent clusters to be visualized, selected, and used for scoring of current or historical data. Each Angle sensor is paired with a node on a distributed computing platform running the Sector middleware. Using Sector, data can be easily transported for analysis or reanalysis.