In traditional cybersecurity, detection is often a post-mortem exercise. Systems collect logs, bundle them into batches, and analyze them hours or even days later. This "batch-processing" paradigm creates a critical window of opportunity for attackers. If a data breach is identified only after the exfiltration is complete, the defense has already failed.
A growing body of research is shifting the focus from retrospective analysis to real-time streaming architectures. The goal is to move detection "to the left" of the attack timeline, identifying and blocking malicious activity while it is still in progress.
The Latency Gap in Batch Processing
The fundamental problem with traditional monitoring is the "latency gap." As Lior Shafir and Avishai Wool point out in A Flow is a Stream of Packets, standard network monitoring requires a conversation to conclude before a summary record is even generated. If an attacker is executing a DDoS attack or exfiltrating data, waiting for the session to end means the damage is already done.
This lag isn't just a network issue; it's systemic. Chaochao Chen and colleagues at Ant Financial found that daily batch processing for fraud detection allows attackers to complete their operations before risk scores can be updated Heterogeneous Graph Neural Networks for Malicious Account Detection. Similarly, in business processes, Gabriel Marques Tavares and Paolo Ceravolo argue that analyzing historical logs prevents organizations from stopping fraudulent transactions as they happen Leveraging Anomaly Detection in Business Process with Data Stream Mining.
From Snapshots to Continuous Streams
To close this gap, researchers are replacing static "snapshots" of data with continuous event streams. This requires a fundamental change in how we model behavior.
Instead of building a graph of an entire day's activity, systems like ORCHID, developed by Akul Goyal and Gang Wang 0011, treat system logs as a "versioned provenance graph" ORCHID: Streaming Threat Detection over Versioned Provenance Graphs. By processing events as they arrive, ORCHID reduces detection lags from 18 hours to sub-millisecond scales.
The intuition here is simple: if you wait for the whole story to be written to check if it's a horror story, the monster has already won. Streaming architectures read the story sentence by sentence, looking for the first sign of trouble.
Lightweight Mechanisms for High-Velocity Data
The challenge of streaming is computational. You cannot run heavy, iterative algorithms on data that is moving at 600,000 queries per second. Researchers have turned to two primary strategies to maintain speed:
- Probabilistic Sketching: In Information-Based Heavy Hitters, Yarin Ozery and Asaf Nadler use "sketches"—mathematical summaries that use tiny amounts of memory—to track DNS data exfiltration in real-time. This allows them to catch attackers stealing data as slowly as 0.7 bytes per second without slowing down the server.
- Frequency Domain Analysis: Krishna Chandra Roy proposes LogSHIELD, which swaps out heavy Graph Neural Networks for lightweight frequency analysis LogSHIELD: A Graph-based Real-time Anomaly Detection Framework using Frequency Analysis. This allows the system to detect zero-day malware in 0.13 seconds, making it viable for live enterprise traffic.
The Shift Toward Online Intervention
The ultimate goal of this research direction is online intervention. This is particularly urgent in new frontiers like LLM security. In Reliable Weak-to-Strong Monitoring of LLM Agents, researchers argue that analyzing an agent's logs after a task is finished cannot prevent immediate harm. Real-time monitoring is required to intercept and block malicious commands before they are executed.
Whether it is Yen‐Yu Chang and Jure Leskovec detecting anomalies in financial "edge streams" F-FADE: Frequency Factorization for Anomaly Detection in Edge Streams or Jalal Ghadermazi identifying malicious intent by the fourth packet of a flow SPIN-IDS, the consensus is clear: the future of security lies in the ability to process and act on data in motion, rather than data at rest.