Adaptive Residual Quantization for Memory-Efficient Temporal Action Segmentation

Gerard Donahue, Guven Gergerli, Ayush Gupta, Reza Ghoddoosian, Enna Sachdeva, Faizan SiddiquiarXiv 2026 · arXiv

TL;DR

ARQ-VAE is a novel, label-free compression framework that slashes the storage requirements for video features in Temporal Action Segmentation (TAS) by orders of magnitude—for example, shrinking a 28 GB dataset to just 10 MB. By using adaptive residual quantization, the model intelligently allocates more memory to complex video transitions and less to redundant, stable scenes, maintaining high downstream performance across supervised, incremental, and unsupervised settings.

What Problem Does This Paper Address?

Modern Temporal Action Segmentation (TAS) systems—which aim to label every frame in long, untrimmed videos—rarely process raw pixels. Instead, they rely on pre-extracted features from heavy-duty backbones like I3D or VideoMAE. While effective, these feature banks are massive, creating a "practical bottleneck" for storage and processing.

This issue is particularly acute in incremental TAS (iTAS). To prevent "catastrophic forgetting," models must store and "replay" features from previous tasks while learning new ones. Existing solutions for this are often rigid, storing data at a fixed segment level, or they require dense, expensive frame-wise labels to guide the compression. The authors identify a need for a label-free, flexible memory representation that can adapt to the varying complexity of video content over time.

Methods

The authors introduce the Adaptive Residual-Quantized Variational AutoEncoder (ARQ-VAE). The core pipeline consists of three main stages:

  1. Residual Vector Quantization (RVQ): A standard RQ-VAE is trained to represent continuous frame features as a chain of discrete codebook indices. Each subsequent stage in the chain quantizes the "residual" (the error) left by the previous stage.
  2. Adaptive Prefix Selection: This is the "secret sauce." Instead of storing a fixed number of codes for every frame, ARQ-VAE evaluates every possible prefix length of the residual chain. It selects the optimal length ($m^*_t$) that minimizes the L2 reconstruction error for that specific frame.
  3. Lightweight Fine-Tuning (ARQ-VAE*): After selecting the optimal prefixes, the encoder is discarded. The decoder and codebooks are then fine-tuned specifically to reconstruct the features from these truncated, variable-length sequences.
The ARQ-VAE architecture uses an encoder to map features to a latent space, quantizes them through multiple residual stages, and decodes them. The stored representation is simply a sequence of discrete indices.

Datasets

The model was evaluated on four standard TAS benchmarks:

  • GTEA: A small egocentric dataset (28 videos).
  • 50Salads: A medium-sized dataset of long procedural videos (50 videos).
  • Breakfast: The largest benchmark, featuring over 1,700 videos of kitchen activities.
  • YouTube Instructional (YTI): Used specifically for evaluating incremental learning settings.

Key Insights and Ideas

  • Complexity is Non-Uniform: Video frames are not created equal. Action transitions or rare visual configurations are "hard" to reconstruct and require deeper residual chains. Conversely, stable interiors of long actions are redundant and can be represented by very short prefixes.
  • Label-Free Utility: Because the compression is based purely on reconstruction error rather than action labels, the same compressed memory can be reused for supervised, unsupervised, and incremental TAS without retraining the compression codec.
  • L2 as a Proxy: The authors demonstrate that minimizing feature-space reconstruction error (L2) is an effective, task-agnostic criterion for selecting prefix lengths, often outperforming label-aware selection methods.
Figure 1: Storage efficiency across TAS settings. ARQ-VAE achieves a better storage–performance trade-off while growing memory more slowly than feature replay and GNI.

Key Results

ARQ-VAE achieves a significantly better storage–performance trade-off than prior state-of-the-art methods like GNI or TCA.

  • Massive Compression: On the Breakfast dataset, ARQ-VAE compressed the original 28 GB I3D feature bank down to just 10 MB—a 2800x reduction.
  • High Fidelity: Despite the extreme compression, the model retained strong performance. Using the ASFormer backbone on the Breakfast dataset, it achieved a 71.5 Edit score and 51.2 F1@50, nearly matching the performance of uncompressed features.
  • Incremental Success: In 10-task incremental learning on Breakfast, ARQ-VAE* outperformed the previous best baseline (GNI), raising the Edit score from 41.1 to 55.7.
Comparison of storage vs. performance across datasets. ARQ-VAE (stars) consistently sits in the high-performance, low-memory quadrant compared to original features and prior baselines.

Limitations and Future Work

While ARQ-VAE is highly effective, the authors note a few limitations:

  • Backbone Dependency: The method compresses pre-extracted features; its ultimate effectiveness is still bounded by the quality of the underlying video backbone (e.g., I3D).
  • Task Specificity: While the mechanism is task-agnostic, the current evidence is focused strictly on TAS.
  • Future Directions: The authors suggest that specialized prefix selection criteria tailored specifically to temporal boundaries could further improve results. They also see potential for applying adaptive residual memories to other areas, such as caching latents for video diffusion models.

Summary updated

Abstract

Temporal action segmentation (TAS) is commonly trained on large banks of pre-extracted frame features, but storing and replaying those features becomes a practical bottleneck for long-form video, especially in incremental TAS where past-task data must remain available to mitigate catastrophic forgetting. Recent TAS replay and condensation methods reduce this burden, but they rely on dense frame-wise annotations and allocate storage through fixed segment-level structures, limiting their flexibility for long procedural videos whose frame complexity varies substantially over time. We introduce Adaptive Residual-Quantized Variational AutoEncoder (ARQ-VAE), a compact and label-free memory representation for TAS features that learns a discrete residual quantization of frame-level video features and then adaptively truncates the residual chain on a per-frame basis, retaining only the residual prefix that best reconstructs each frame feature. This yields a practical compact feature memory that does not depend on frame-wise labels and can therefore support multiple TAS settings with the same stored representation, while a lightweight fine-tuning stage further improves reconstruction quality for the truncated codes that are actually retained. Across standard TAS benchmarks, ARQ-VAE achieves a substantially improved storage-performance trade-off over prior replay and condensation baselines. On Breakfast, our final representation reduces training-set storage from 28 GB for the original I3D feature bank to 10 MB while retaining strong downstream segmentation performance; with ASFormer, it achieves 71.5 Edit and 51.2 F1@50 in the supervised setting. We further show that the same label-free memory remains effective for incremental and unsupervised TAS. We position ARQ-VAE as a practical memory representation for TAS and a promising direction for broader dataset condensation applications.
BibTeX
@misc{donahue2026adaptive,
  author = {Gerard Donahue and Guven Gergerli and Ayush Gupta and Reza Ghoddoosian and Enna Sachdeva and Faizan Siddiqui},
  title = {{Adaptive Residual Quantization for Memory-Efficient Temporal Action Segmentation}},
  year = {2026},
  eprint = {2611.13262},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.13262},
}
Figure