Traditional data mining and process analysis operate under a "post-mortem" paradigm. Systems collect data over weeks or months, write it to static log files, and analyze it retrospectively to map out what happened. While this approach provides valuable historical insights, it introduces significant latency. By the time an analyst discovers a bottleneck, a structural deviation, or an emerging trend, the operational reality has already moved on.
To bridge this gap, a major research direction has emerged focused on incremental, real-time discovery from continuous event streams. Instead of treating logs as static repositories, this paradigm treats data as an endless, high-velocity stream of events. The goal is to construct, update, and refine structural models—such as business process maps, system log templates, or behavioral trends—on the fly, without ever reprocessing historical data.
The Shift from Post-Mortem to Live Models
In a big data environment, waiting for batch processes is often unfeasible. As Antonia Azzini outlines in Process Mining in Big Data Scenario, transitioning from static log files to continuous event streams allows organizations to maintain a live representation of their operational state. Rather than capturing a frozen snapshot of the past, live models adapt dynamically as new digital traces arrive.
This real-time visibility is critical for proactive intervention. For instance, Williams Rizzi et al. demonstrate in A ProM Operational Support Provider for Predictive Monitoring of Business Processes that ingesting live event streams enables predictive monitoring. Instead of discovering that a workflow failed after the fact, systems can evaluate active, incomplete cases against the evolving model to trigger immediate alerts or operational adjustments.
However, moving from batch processing to streaming processing introduces a fundamental algorithmic challenge: How do you build and maintain a coherent model when you cannot store the entire dataset or look backward in time?
Bounding Memory via Incremental State Maintenance
The core mechanism behind online discovery is the maintenance of an abstracted, bounded state. Because an event stream is theoretically infinite, algorithms cannot retain every raw event. Instead, they must update structural summaries incrementally.
In the context of control-flow discovery, Andrea Burattin et al. address this in Control-flow discovery from event streams by adapting traditional process mining to the streaming domain. Rather than re-mining a model from scratch—which is computationally prohibitive—their approach maintains real-time frequency counts of activities and direct successions (i.e., how often event $B$ immediately follows event $A$).
To prevent memory consumption from growing indefinitely, the authors utilize two primary strategies:
- Sliding Windows: Only events within a recent temporal or numerical window contribute to the model's state, causing older behavior to naturally drop out.
- Lossy Counting: A mathematical approximation technique that discards low-frequency activity successions over time, ensuring that only statistically significant pathways are preserved in memory.
[Incoming Event Stream] ---> [Real-time Frequency Counter] ---> [Lossy Counting / Windowing] ---> [Updated Process Model]
(Tracks A -> B, B -> C) (Prunes infrequent paths)
By managing state this way, the system can output an updated control-flow model instantly upon the arrival of each new event, achieving the low latency required for live monitoring.
Adapting to Change and Concept Drift
In real-world streams, the underlying processes are rarely stationary. A business might change its compliance rules, a website might update its user interface, or public interest might shift overnight. This phenomenon is known as concept drift.
If an online discovery system relies on a rigid, pre-defined template, it quickly becomes obsolete. As Davide Mora et al. explain in The CDESF Toolkit: An Introduction, an effective streaming framework must feature incremental learning capabilities that allow reference models to evolve alongside actual behavior. By deploying online clustering techniques, their framework dynamically detects when incoming event sequences deviate so fundamentally from historical patterns that they constitute a permanent shift in the underlying process, rather than a temporary anomaly.
This challenge of separating structural evolution from temporary noise appears across multiple domains:
- Process Clustering: Florian Richter 0001 et al. note in k-process: Model-Conformance-based Clustering of Process Instances that online scenarios require models to identify entirely new process variants or behavioral deviations on the fly, rather than fitting data into static, pre-allocated clusters.
- Time-Series Regimes: In Fast and Multi-aspect Mining of Complex Time-stamped Event Streams, Kota Nakamura et al. introduce a method called CUBESCOPE, which continuously processes high-dimensional event streams to identify distinct temporal "regimes" and latent components, updating its structural understanding without reprocessing historical data.
- Stochastic Anomaly Detection: For spatio-temporal streams, Shixiang Zhu et al. demonstrate in Adversarial Anomaly Detection for Marked Spatio-Temporal Streaming Data how combining recurrent neural networks with generative adversarial frameworks allows a system to evaluate incoming data points incrementally against a learned stochastic process, triggering immediate alerts when a sequence drifts from expected behavior.
Structuring Unstructured Streams
A significant hurdle to streaming discovery is that real-world event streams are rarely clean or well-structured. Often, they arrive as raw text or low-level system messages that must be abstracted before any high-level process discovery can occur.
In system telemetry, for example, logs are generated as free-form text strings. As Han Qiu et al. survey in System Log Parsing: A Survey, online log parsing requires algorithms to learn and update log templates dynamically as messages stream in. Instead of relying on a fixed dictionary of known log formats, the parser must recognize constants and variables on the fly, adapting to new software versions or log formats without needing a complete re-processing of historical data.
[Raw Log Stream] ------------------------> [Online Parser] ------------------------> [Downstream Discovery]
"Connection failed from 192.168.1.1" Learns template: Maps high-level system
"Connection failed from 10.0.0.5" "Connection failed from <*>" behaviors in real time
Similarly, raw event logs often contain low-level "spaghetti" details that obscure the broader business process. Adrian Rebmann et al. argue in GECCO: Constraint-driven Abstraction of Low-level Event Logs that transitioning event abstraction to online environments is a critical necessity. To make streaming discovery useful, systems need low-latency mechanisms that can group these messy, low-level events into high-level, meaningful activities in real time.
Cross-Domain Parallelisms: Text and Social Streams
The mathematical and conceptual frameworks powering streaming process discovery closely parallel developments in text and social media stream mining. In these domains, the challenge is also to extract structured, evolving relationships from an endless influx of noisy data.
For example, in natural language processing, Congle Zhang et al. develop an unsupervised system in Exploiting Parallel News Streams for Unsupervised Event Extraction to extract fluent event relations from live news feeds. They emphasize that moving away from batch processing is essential for real-time information monitoring.
This sentiment is echoed in event detection frameworks for social media. Tao Ge et al. present a time-aware architecture in Event Detection with Burst Information Networks that models words as temporal "bursts" within a continuous text stream, avoiding retrospective document clustering in favor of live monitoring. To separate these short-lived, bursty spikes from long-term underlying patterns, Tijl De Bie et al. demonstrate in Detecting trends in twitter time series how optimization frameworks can be adapted from retrospective batch analysis into real-time processing tools to power early warning systems.
Summary
Whether tracking a business process, monitoring a distributed server cluster, or analyzing global news feeds, the core philosophy of incremental stream discovery remains the same: the model must live inside the stream. By maintaining bounded frequency representations, utilizing lossy counting or sliding windows, and deploying online clustering for concept drift, modern architectures successfully eliminate the latency of batch processing. They transform data analysis from a retrospective look at the past into an active, evolving map of the present.
Key Foundational Work
Process Mining in Big Data Scenario (2015) — contributed 2 concept elements to this direction.
A ProM Operational Support Provider for Predictive Monitoring of Business Processes (2015) — contributed 1 concept elements to this direction.
Exploiting Parallel News Streams for Unsupervised Event Extraction (2015) — contributed 1 concept elements to this direction.