TL;DR
SnapLog is a lightweight, privacy-preserving pipeline that automatically extracts structured event logs from raw video streams for process mining. By combining unsupervised temporal action segmentation with few-shot video classification, it bypasses the need for massive labeled datasets or expensive, cloud-dependent Visual Language Models (VLMs).
What Problem Does This Paper Address?
Process mining is a powerful discipline that helps organizations analyze, discover, and improve real-world workflows by looking at digital footprints called event logs (which contain activity labels, timestamps, and case IDs). However, in many physical environments—like hospitals, factories, or airports—workflows aren't captured in neat databases; they happen in front of security and surveillance cameras.
Turning raw video into structured event logs is a massive bottleneck. Manual annotation doesn't scale, fully supervised deep learning requires thousands of labeled clips, and recent trends using massive foundation models (like GPT-4V cascades) are computationally expensive, slow, and raise serious data privacy concerns for sensitive organizational data.
This paper introduces SnapLog, a resource-constrained, on-premise alternative designed to bridge the gap between high-dimensional video data and standard process mining tools.
Methods
SnapLog breaks the video-to-log problem into two main stages:
- Temporal Action Segmentation (TAS): To find out when activities start and end without any prior labels, SnapLog processes video frames independently using a pre-trained Vision Transformer (ViT) to generate frame-level embeddings. It then computes a frame-wise cosine similarity matrix, contextualized with temporal indices to ensure time-adjacent frames cluster together. K-Means clustering identifies "atomic events," which are then refined by a greedy merging algorithm to eliminate over-fragmentation based on a minimum duration threshold ($l_{min}$).
- Few-Shot Video Classification: Once the video is segmented into discrete clips, an R(2+1)D convolutional network (pre-trained on the Sports1M dataset) extracts spatiotemporal features. To adapt to new, domain-specific tasks with very few examples (<20 samples), the authors freeze the R(2+1)D backbone and train a lightweight linear classification head using cross-entropy loss and heavy data augmentation (rotations, brightness scaling, noise).
- Uncertainty-Aware Log Abstraction: Instead of forcing a deterministic "hard" label on ambiguous video segments, SnapLog can output the probability distribution over candidate activities. This yields an uncertain event log, preserving semantic ambiguity for downstream uncertainty-aware process mining algorithms.
Datasets
The authors evaluated SnapLog on two distinct real-world video datasets:
- TUM Kitchen Dataset: 16 videos of a brownie-baking process in a stable environment with a consistent background. The authors used an LLM to aggregate low-level annotations (e.g., "take spoon") into 16 compound activities suitable for process mining (e.g., "prepare batter").
- Epic Kitchens-100 Dataset: 24 "in-the-wild" egocentric (first-person) videos focusing on kitchen cleaning across 10 different environments. This dataset provided a rigorous test for robustness against shifting backgrounds and camera angles.
Key Insights and Ideas
- Embracing Ambiguity: Video data is inherently uncertain. A clip of someone standing near an oven could mean "preheating" or "inserting food." SnapLog’s ability to output probabilistic logs prevents premature, incorrect data decisions, allowing process mining tools to handle the ambiguity natively.
- Lean and Local: Unlike heavy VLM/LLM pipelines that require massive cloud compute clusters, SnapLog runs efficiently on standard local hardware with a single GPU. This makes it highly scalable for large video archives and compliant with strict corporate privacy policies.
- Temporal Context Matters: Simply clustering raw frame embeddings often fails because the same visual scene can reappear at different times. Appending the frame index to the similarity matrix before clustering stabilizes the temporal boundaries, successfully separating repeated actions into distinct events.
Key Results
The pipeline achieves strong performance even in extreme low-data regimes:
- Segmentation Accuracy: The unsupervised segmentation achieved a frame-wise accuracy of 74.3% on the TUM dataset and 67.7% on the more complex Epic Kitchens dataset.
- Classification Performance: When utilizing an overlapping clip-sampling strategy combined with spatial augmentations, the few-shot classifier achieved a Top-3 accuracy of 90.1% for TUM and 85.4% for Epic Kitchens.
- Process Discovery: Downstream process mining tools successfully reconstructed a highly accurate, readable workflow model from the generated logs, capturing the correct logical order of steps with only minor ordering inconsistencies.
Limitations and Future Work
While SnapLog represents a major step toward practical video-based process mining, the authors note a few limitations:
- Single-Activity Assumption: The model assumes only one activity happens at a time within a given segment. It cannot currently handle complex scenes where multiple independent events occur simultaneously.
- Loss of Spatial Data: SnapLog focuses entirely on what is happening and when. It discards spatial information (like bounding boxes or coordinates), which could otherwise be used to optimize physical layouts or calculate transportation distances in environments like manufacturing lines.
- Human-in-the-Loop: While labeling requirements are low, it still requires initial human annotations for the few-shot adaptation phase. Future work may explore using VLMs to automate this initial seeding process.
Summary updated