DUGO: Dual-Scale Gaussian Memory for Online Temporal Action Localization in Egocentric Procedural Videos

Mohammed Aman Bhuiyan, Md. Sazzad Hossain Adib, Samiul Basir Bhuiyan, Mir Faiyaz Hossain, Syed Mohammed Shamsul Islam, Md Zakir Hossain, Shafin RahmanarXiv 2026 · arXiv

TL;DR

DUGO (Dual-Scale Gaussian Memory Online) is a new framework for real-time action detection in streaming videos, specifically designed for complex, multi-step tasks like cooking. By combining a dual-scale encoder for fine-grained motion and a stochastic Gaussian memory bank to handle historical ambiguity, it achieves state-of-the-art results on major egocentric and procedural benchmarks.

What Problem Does This Paper Address?

Temporal Action Localization (TAL) involves identifying the "what" and "when" of actions in a video. While offline models can look at a whole video to refine their guesses, Online Temporal Action Localization (OnTAL) must work in real-time on a live stream. It cannot see the future, and it cannot change a prediction once it has been committed.

This is particularly difficult in procedural egocentric videos (first-person views of tasks like assembly or cooking). These videos present three main challenges:

  1. Fine-grained discrimination: Actions like "open" and "close" look very similar and require high-resolution local motion cues.
  2. Long-range dependencies: Understanding that a "cut" action is happening often requires remembering that a "wash" action happened minutes ago.
  3. Procedural Ambiguity: History isn't always a straight line; deterministic memory often fails to capture the multi-modal nature of complex tasks.
Motivation: Egocentric procedural videos require fine-grained motion discrimination and memory of prior steps.

Methods

The authors introduce DUGO, which moves away from single-scale, deterministic models toward a more flexible, dual-pronged architecture.

1. Dual-Scale Temporal Encoder (DSE)

Standard Transformers process entire windows of time uniformly, which can wash out sharp, brief signals like a sudden wrist rotation. The DSE solves this by using two parallel branches:

  • Local Branch: Uses depth-wise 1D convolutions to capture per-channel dynamics and fine-grained motion patterns.
  • Global Branch: Uses a Transformer encoder to model structural dependencies across the entire observation window.

2. Gaussian Memory Bank (GMB)

Instead of compressing history into a single fixed vector (which loses information), DUGO represents memory as a stochastic mixture of Gaussian prototypes. This allows the model to maintain a distribution of possible past actions, preserving the "multi-modal ambiguity" inherent in procedural tasks. This memory is injected twice: once to refine the encoder's memory and again to enhance the final anchor queries.

3. 1D Distance-IoU (DIoU) Loss

To improve the precision of action boundaries, the authors adapted the DIoU loss (common in 2D object detection) for 1D temporal segments. This loss doesn't just look at overlap; it specifically penalizes the distance between the centers of the predicted and ground-truth segments.

Fig. 2: DUGO Architecture. DSE replaces the single-granularity encoder with dual branches consisting of depthwise temporal convolutions for local modeling and self-attention for global temporal reasoning, followed by fusion through a learned projection. GMB replaces deterministic history compression with a stochastic mixture of Gaussian prototypes injected at two stages: encoder memory refinement (Pass 1) and decoded anchor query enhancement (Pass 2) through per-head gated cross-attention.

Datasets

The model was evaluated on three major benchmarks:

  • EGTEA: A procedural egocentric dataset of kitchen activities (28 hours).
  • EPIC-Kitchens-100: A large-scale egocentric benchmark with nearly 100 hours of video and 97 verb classes.
  • MUSES: A non-egocentric dataset consisting of TV shows and movies, used to test the model's generalization to less procedural content.

Key Insights and Ideas

  • Stochastic > Deterministic: The biggest "aha" moment is that procedural history is better represented as a distribution than a point. By using Gaussian prototypes, the model handles the uncertainty of "what happened before" much more effectively than prior methods.
  • Local Cues Matter: In egocentric video, the difference between success and failure often lies in a few frames of hand movement. The depth-wise convolution branch in the DSE ensures these signals aren't lost in the global self-attention mechanism.
  • Twice-Injected Memory: The ablation studies showed that injecting memory at both the encoder and decoder stages provides complementary benefits, significantly boosting performance over single-injection methods.

Key Results

DUGO established a new state-of-the-art across all tested benchmarks:

  • EGTEA: Achieved an average mAP of 24.3%, outperforming the previous best (HAT) by 2.8%.
  • EPIC-Kitchens-100: Reached 16.6% mAP, a 1.3% improvement over HAT.
  • Efficiency: Despite its complexity, DUGO runs at 40.2 FPS on a single consumer GPU (RTX 3060 Ti), making it well-suited for real-time deployment. It also uses significantly fewer parameters (129M) compared to HAT (247M).
TABLE I: Comparison of mAP (%) across tIoU thresholds on EGTEA and EPIC-Kitchens-100 datasets. ∗Indicates best result.

Limitations and Future Work

The authors note that the current Gaussian Memory Bank only adapts prototypes based on the current short-term encoder window. This limits the model's ability to perform persistent, long-horizon reasoning over very repetitive or extremely long activities.

Future work will look into recurrent step-aware memory and incorporating object-centric representations (e.g., tracking the state of a knife or a pan) to further improve procedural understanding.

Summary updated

Abstract

Online Temporal Action Localization (OnTAL) aims to detect and localize action instances from untrimmed streaming videos in real time. Unlike offline approaches, OnTAL models have no access to future frames and cannot go back and refine earlier predictions, which makes the task considerably harder given the strict real-time constraints and narrow temporal window available. Most existing OnTAL methods lean on either brief anchor-based context or a single-scale memory of past frames, which limits how well they can capture actions unfolding across multiple time scales or requiring long-range reasoning in Procedural Videos. To address these gaps, we present DUGO (Dual-Scale Encoding and Gaussian Memory Online), a new OnTAL framework that jointly models local motion dynamics and long-range temporal dependencies. At its core, DUGO uses a Dual-Scale Temporal Encoder, pairing depth-wise local representations with global self-attention so it can pick up both fine-grained temporal patterns and broader contextual cues. We also introduce a Gaussian Memory Bank, which represents action memory as stochastic, input-conditioned Gaussian prototypes, allowing for richer temporal memory through gated cross-attention applied at both the encoder and anchor-query stages. Extensive experiments on procedural egocentric datasets (EGTEA, EPIC-Kitchens-100) demonstrate that DUGO achieves state-of-the-art performance while also providing substantial improvements on non-egocentric benchmarks such as Muses. The project page is available at: https://aman554-eq.github.io/DUGO
BibTeX
@misc{bhuiyan2026dugo,
  author = {Mohammed Aman Bhuiyan and Md. Sazzad Hossain Adib and Samiul Basir Bhuiyan and Mir Faiyaz Hossain and Syed Mohammed Shamsul Islam and Md Zakir Hossain and Shafin Rahman},
  title = {{DUGO: Dual-Scale Gaussian Memory for Online Temporal Action Localization in Egocentric Procedural Videos}},
  year = {2026},
  eprint = {2610.41402},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.41402},
}
Figure