All Eyes on the Workflow: Automated and Efficient Event Discovery from Video Streams

Marco Pegoraro, Jonas Seng, Dustin Heller, Wil M. P. van der Aalst, Kristian KerstingarXiv 2026 · arXiv

TL;DR

SnapLog is a lightweight, privacy-preserving pipeline that automatically extracts structured event logs from raw video streams for process mining. By combining unsupervised temporal action segmentation with few-shot video classification, it bypasses the need for massive labeled datasets or expensive, cloud-dependent Visual Language Models (VLMs).

What Problem Does This Paper Address?

Process mining is a powerful discipline that helps organizations analyze, discover, and improve real-world workflows by looking at digital footprints called event logs (which contain activity labels, timestamps, and case IDs). However, in many physical environments—like hospitals, factories, or airports—workflows aren't captured in neat databases; they happen in front of security and surveillance cameras.

Turning raw video into structured event logs is a massive bottleneck. Manual annotation doesn't scale, fully supervised deep learning requires thousands of labeled clips, and recent trends using massive foundation models (like GPT-4V cascades) are computationally expensive, slow, and raise serious data privacy concerns for sensitive organizational data.

This paper introduces SnapLog, a resource-constrained, on-premise alternative designed to bridge the gap between high-dimensional video data and standard process mining tools.

Fig. 1. SnapLog extracts logs from video data. Our method, SnapLog, constructs an event log from video data which can then be used downstream by standard process mining algorithms.

Methods

SnapLog breaks the video-to-log problem into two main stages:

  1. Temporal Action Segmentation (TAS): To find out when activities start and end without any prior labels, SnapLog processes video frames independently using a pre-trained Vision Transformer (ViT) to generate frame-level embeddings. It then computes a frame-wise cosine similarity matrix, contextualized with temporal indices to ensure time-adjacent frames cluster together. K-Means clustering identifies "atomic events," which are then refined by a greedy merging algorithm to eliminate over-fragmentation based on a minimum duration threshold ($l_{min}$).
Fig. 2. Video Segmentation. From left to right: After the extraction of frame-wise embeddings using a ViT, the embeddings are used to compute a similarity matrix. By applying K-Means on the similarity matrix, atomic events are formed. These are merged into larger action segments using a greedy merging algorithm that pairs up atomic events based on their proximity in the embedding space, thus achieving a semantically grounded grouping.
  1. Few-Shot Video Classification: Once the video is segmented into discrete clips, an R(2+1)D convolutional network (pre-trained on the Sports1M dataset) extracts spatiotemporal features. To adapt to new, domain-specific tasks with very few examples (<20 samples), the authors freeze the R(2+1)D backbone and train a lightweight linear classification head using cross-entropy loss and heavy data augmentation (rotations, brightness scaling, noise).
  2. Uncertainty-Aware Log Abstraction: Instead of forcing a deterministic "hard" label on ambiguous video segments, SnapLog can output the probability distribution over candidate activities. This yields an uncertain event log, preserving semantic ambiguity for downstream uncertainty-aware process mining algorithms.

Datasets

The authors evaluated SnapLog on two distinct real-world video datasets:

  • TUM Kitchen Dataset: 16 videos of a brownie-baking process in a stable environment with a consistent background. The authors used an LLM to aggregate low-level annotations (e.g., "take spoon") into 16 compound activities suitable for process mining (e.g., "prepare batter").
  • Epic Kitchens-100 Dataset: 24 "in-the-wild" egocentric (first-person) videos focusing on kitchen cleaning across 10 different environments. This dataset provided a rigorous test for robustness against shifting backgrounds and camera angles.

Key Insights and Ideas

  • Embracing Ambiguity: Video data is inherently uncertain. A clip of someone standing near an oven could mean "preheating" or "inserting food." SnapLog’s ability to output probabilistic logs prevents premature, incorrect data decisions, allowing process mining tools to handle the ambiguity natively.
  • Lean and Local: Unlike heavy VLM/LLM pipelines that require massive cloud compute clusters, SnapLog runs efficiently on standard local hardware with a single GPU. This makes it highly scalable for large video archives and compliant with strict corporate privacy policies.
  • Temporal Context Matters: Simply clustering raw frame embeddings often fails because the same visual scene can reappear at different times. Appending the frame index to the similarity matrix before clustering stabilizes the temporal boundaries, successfully separating repeated actions into distinct events.

Key Results

The pipeline achieves strong performance even in extreme low-data regimes:

  • Segmentation Accuracy: The unsupervised segmentation achieved a frame-wise accuracy of 74.3% on the TUM dataset and 67.7% on the more complex Epic Kitchens dataset.
  • Classification Performance: When utilizing an overlapping clip-sampling strategy combined with spatial augmentations, the few-shot classifier achieved a Top-3 accuracy of 90.1% for TUM and 85.4% for Epic Kitchens.
  • Process Discovery: Downstream process mining tools successfully reconstructed a highly accurate, readable workflow model from the generated logs, capturing the correct logical order of steps with only minor ordering inconsistencies.

Limitations and Future Work

While SnapLog represents a major step toward practical video-based process mining, the authors note a few limitations:

  • Single-Activity Assumption: The model assumes only one activity happens at a time within a given segment. It cannot currently handle complex scenes where multiple independent events occur simultaneously.
  • Loss of Spatial Data: SnapLog focuses entirely on what is happening and when. It discards spatial information (like bounding boxes or coordinates), which could otherwise be used to optimize physical layouts or calculate transportation distances in environments like manufacturing lines.
  • Human-in-the-Loop: While labeling requirements are low, it still requires initial human annotations for the few-shot adaptation phase. Future work may explore using VLMs to automate this initial seeding process.

Summary updated

Abstract

Disciplines such as business process management and process mining aid organizations by discovering insights about processes on the basis of recorded event data. However, an obstacle to process analysis is data multi-modality: for instance, data in video form are not directly interpretable as events. Existing approaches rely on a dictionary of activity label as input, cannot provide frame-by-frame labeling explanations, or rely on superseded computer vision techniques. In this work, we present SnapLog, an approach to extract event data from videos by converting frames to feature vectors using image embeddings and performing temporal segmentation through frame-wise similarity matrices. A generalized few-shot classification is then used to assign labels to the video segments, yielding labeled, timestamped sub-sequences of frames that are interpretable as events. Conventional process mining techniques can be used to analyze the resulting data. We show that our approach produces logs that accurately reflect the process in the videos.
BibTeX
@misc{pegoraro2026all,
  author = {Marco Pegoraro and Jonas Seng and Dustin Heller and Wil M. P. van der Aalst and Kristian Kersting},
  title = {{All Eyes on the Workflow: Automated and Efficient Event Discovery from Video Streams}},
  year = {2026},
  eprint = {2604.22476},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.22476},
}
Figure