Data Analysis Challenges

Abstract

JASON was asked to recommend ways in which the DOD/IC can handle present and future sensor data in fundamentally different ways, taking into account both the state-of-the-art, the potential for advances in areas such as data structures, the shaping of sensor data for exploitation, as well as methodologies for data discovery. This report examines the challenges associated with the analysis of large data and in particular compares DOD/IC requirements to those of several data intensive fields. JASON finds that DOD/IC data requirements are certainly significant, but not unmanageable given the capabilities of current and projected storage technology. The key challenge will be to adequately empower the analyst by matching analysis needs to data delivery modalities. The report also proposes various grand challenges that could be used to assess and prioritize future research efforts in data assimilation and fusion.