TL;DR
ARQ-VAE is a novel, label-free compression framework that slashes the storage requirements for video features in Temporal Action Segmentation (TAS) by orders of magnitude—for example, shrinking a 28 GB dataset to just 10 MB. By using adaptive residual quantization, the model intelligently allocates more memory to complex video transitions and less to redundant, stable scenes, maintaining high downstream performance across supervised, incremental, and unsupervised settings.
What Problem Does This Paper Address?
Modern Temporal Action Segmentation (TAS) systems—which aim to label every frame in long, untrimmed videos—rarely process raw pixels. Instead, they rely on pre-extracted features from heavy-duty backbones like I3D or VideoMAE. While effective, these feature banks are massive, creating a "practical bottleneck" for storage and processing.
This issue is particularly acute in incremental TAS (iTAS). To prevent "catastrophic forgetting," models must store and "replay" features from previous tasks while learning new ones. Existing solutions for this are often rigid, storing data at a fixed segment level, or they require dense, expensive frame-wise labels to guide the compression. The authors identify a need for a label-free, flexible memory representation that can adapt to the varying complexity of video content over time.
Methods
The authors introduce the Adaptive Residual-Quantized Variational AutoEncoder (ARQ-VAE). The core pipeline consists of three main stages:
- Residual Vector Quantization (RVQ): A standard RQ-VAE is trained to represent continuous frame features as a chain of discrete codebook indices. Each subsequent stage in the chain quantizes the "residual" (the error) left by the previous stage.
- Adaptive Prefix Selection: This is the "secret sauce." Instead of storing a fixed number of codes for every frame, ARQ-VAE evaluates every possible prefix length of the residual chain. It selects the optimal length ($m^*_t$) that minimizes the L2 reconstruction error for that specific frame.
- Lightweight Fine-Tuning (ARQ-VAE*): After selecting the optimal prefixes, the encoder is discarded. The decoder and codebooks are then fine-tuned specifically to reconstruct the features from these truncated, variable-length sequences.
Datasets
The model was evaluated on four standard TAS benchmarks:
- GTEA: A small egocentric dataset (28 videos).
- 50Salads: A medium-sized dataset of long procedural videos (50 videos).
- Breakfast: The largest benchmark, featuring over 1,700 videos of kitchen activities.
- YouTube Instructional (YTI): Used specifically for evaluating incremental learning settings.
Key Insights and Ideas
- Complexity is Non-Uniform: Video frames are not created equal. Action transitions or rare visual configurations are "hard" to reconstruct and require deeper residual chains. Conversely, stable interiors of long actions are redundant and can be represented by very short prefixes.
- Label-Free Utility: Because the compression is based purely on reconstruction error rather than action labels, the same compressed memory can be reused for supervised, unsupervised, and incremental TAS without retraining the compression codec.
- L2 as a Proxy: The authors demonstrate that minimizing feature-space reconstruction error (L2) is an effective, task-agnostic criterion for selecting prefix lengths, often outperforming label-aware selection methods.
Key Results
ARQ-VAE achieves a significantly better storage–performance trade-off than prior state-of-the-art methods like GNI or TCA.
- Massive Compression: On the Breakfast dataset, ARQ-VAE compressed the original 28 GB I3D feature bank down to just 10 MB—a 2800x reduction.
- High Fidelity: Despite the extreme compression, the model retained strong performance. Using the ASFormer backbone on the Breakfast dataset, it achieved a 71.5 Edit score and 51.2 F1@50, nearly matching the performance of uncompressed features.
- Incremental Success: In 10-task incremental learning on Breakfast, ARQ-VAE* outperformed the previous best baseline (GNI), raising the Edit score from 41.1 to 55.7.
Limitations and Future Work
While ARQ-VAE is highly effective, the authors note a few limitations:
- Backbone Dependency: The method compresses pre-extracted features; its ultimate effectiveness is still bounded by the quality of the underlying video backbone (e.g., I3D).
- Task Specificity: While the mechanism is task-agnostic, the current evidence is focused strictly on TAS.
- Future Directions: The authors suggest that specialized prefix selection criteria tailored specifically to temporal boundaries could further improve results. They also see potential for applying adaptive residual memories to other areas, such as caching latents for video diffusion models.
Summary updated