TL;DR
DUGO (Dual-Scale Gaussian Memory Online) is a new framework for real-time action detection in streaming videos, specifically designed for complex, multi-step tasks like cooking. By combining a dual-scale encoder for fine-grained motion and a stochastic Gaussian memory bank to handle historical ambiguity, it achieves state-of-the-art results on major egocentric and procedural benchmarks.
What Problem Does This Paper Address?
Temporal Action Localization (TAL) involves identifying the "what" and "when" of actions in a video. While offline models can look at a whole video to refine their guesses, Online Temporal Action Localization (OnTAL) must work in real-time on a live stream. It cannot see the future, and it cannot change a prediction once it has been committed.
This is particularly difficult in procedural egocentric videos (first-person views of tasks like assembly or cooking). These videos present three main challenges:
- Fine-grained discrimination: Actions like "open" and "close" look very similar and require high-resolution local motion cues.
- Long-range dependencies: Understanding that a "cut" action is happening often requires remembering that a "wash" action happened minutes ago.
- Procedural Ambiguity: History isn't always a straight line; deterministic memory often fails to capture the multi-modal nature of complex tasks.
Methods
The authors introduce DUGO, which moves away from single-scale, deterministic models toward a more flexible, dual-pronged architecture.
1. Dual-Scale Temporal Encoder (DSE)
Standard Transformers process entire windows of time uniformly, which can wash out sharp, brief signals like a sudden wrist rotation. The DSE solves this by using two parallel branches:
- Local Branch: Uses depth-wise 1D convolutions to capture per-channel dynamics and fine-grained motion patterns.
- Global Branch: Uses a Transformer encoder to model structural dependencies across the entire observation window.
2. Gaussian Memory Bank (GMB)
Instead of compressing history into a single fixed vector (which loses information), DUGO represents memory as a stochastic mixture of Gaussian prototypes. This allows the model to maintain a distribution of possible past actions, preserving the "multi-modal ambiguity" inherent in procedural tasks. This memory is injected twice: once to refine the encoder's memory and again to enhance the final anchor queries.
3. 1D Distance-IoU (DIoU) Loss
To improve the precision of action boundaries, the authors adapted the DIoU loss (common in 2D object detection) for 1D temporal segments. This loss doesn't just look at overlap; it specifically penalizes the distance between the centers of the predicted and ground-truth segments.
Datasets
The model was evaluated on three major benchmarks:
- EGTEA: A procedural egocentric dataset of kitchen activities (28 hours).
- EPIC-Kitchens-100: A large-scale egocentric benchmark with nearly 100 hours of video and 97 verb classes.
- MUSES: A non-egocentric dataset consisting of TV shows and movies, used to test the model's generalization to less procedural content.
Key Insights and Ideas
- Stochastic > Deterministic: The biggest "aha" moment is that procedural history is better represented as a distribution than a point. By using Gaussian prototypes, the model handles the uncertainty of "what happened before" much more effectively than prior methods.
- Local Cues Matter: In egocentric video, the difference between success and failure often lies in a few frames of hand movement. The depth-wise convolution branch in the DSE ensures these signals aren't lost in the global self-attention mechanism.
- Twice-Injected Memory: The ablation studies showed that injecting memory at both the encoder and decoder stages provides complementary benefits, significantly boosting performance over single-injection methods.
Key Results
DUGO established a new state-of-the-art across all tested benchmarks:
- EGTEA: Achieved an average mAP of 24.3%, outperforming the previous best (HAT) by 2.8%.
- EPIC-Kitchens-100: Reached 16.6% mAP, a 1.3% improvement over HAT.
- Efficiency: Despite its complexity, DUGO runs at 40.2 FPS on a single consumer GPU (RTX 3060 Ti), making it well-suited for real-time deployment. It also uses significantly fewer parameters (129M) compared to HAT (247M).
Limitations and Future Work
The authors note that the current Gaussian Memory Bank only adapts prototypes based on the current short-term encoder window. This limits the model's ability to perform persistent, long-horizon reasoning over very repetitive or extremely long activities.
Future work will look into recurrent step-aware memory and incorporating object-centric representations (e.g., tracking the state of a knife or a pan) to further improve procedural understanding.
Summary updated