Nancy J. McMillan, Douglas D. Mooney, and David A. Burgoon (2003), Continually Improving Stream Analysis for Network Security, Computing Science and Statistics, 35, I2003Proceedings/McMillanNancy/McMillanNancy.paper.pdf
Nancy J. McMillan, Douglas D. Mooney, and David A. Burgoon (2003), CISA - Continually Improving Stream Analysis, Computing Science and Statistics, 35, I2003Proceedings/McMillanNancy/McMillanNancy.presentation.pdf ,
Other files: McMillanNancy.Example1.pdf, McMillanNancy.Example1.ppt,
In many real-world environments, events happen at irregular intervals and measurements describing those events are recorded, e.g., network connection attempts. The flow of measurements thus generated is stream data. The pace of real world events, which is not controlled, governs the rate at which stream data flows. The real-time management and use of stream data for decision-making and/or characterization is complicated by the variable flow rate inherent in this data. By nature, these activities require data management and data processing based on algorithms. Data management and processing takes time; the amount of time is governed by the complexity of the algorithms employed. Typically more complex and time-consuming algorithms are only considered when they provide superior decision-making ability or superior characterization. However, if new data cannot be handled as quickly as it is generated, real-time management and use is not occurring. Thus, there is a natural trade-off between algorithms that store and process data quickly enough to keep up with the flow of stream data and algorithms that provide a sufficiently accurate decision or characterization. Continually improving stream analysis (CISA) is proposed as a mechanism for managing the trade-off between providing sufficiently accurate decisions/characterizations and keeping up with the flow of stream data. The real-time stream data monitoring features that are provided by CISA are: 1. The algorithm always provides a current decision or characterization. 2. The precision/accuracy of the current decision or characterization improves when there is more processing time available relative to the rate of data flow, i.e., more processors, faster processors, and slower data flow all translate to more precise/accurate decisions or characterizations. 3. The algorithm scales automatically to optimize accuracy/precision of the current decision or characterization as a function of data flow rate. In this work, concepts for CISA are realized in the framework of a cyber security example. Specifically, a CISA intruder detection system (IDS) is developed, which monitors firewall data. The IDS developed is a dynamic classification/characterization tool that first identifies groups of sources by common behavior patterns then characterizes the behavior of the groups identified over time. As new intruder behaviors emerge, they are captured by the appearance of new groups or the migration in behavior patterns of existing groups.