TL;DR
SAVA-X introduces a novel framework for Ego$\rightarrow$Exo Imitation Error Detection, identifying procedural mistakes when a first-person (egocentric) execution attempts to copy a third-person (exocentric) demonstration. By combining adaptive frame sampling, scene-aware view embeddings, and bidirectional cross-view fusion, the model successfully overcomes massive temporal misalignments and domain shifts to accurately localize and flag execution errors.
What Problem Does This Paper Address?
Imagine a student nurse trying to replicate a complex medical protocol after watching an instructor's demonstration, or a factory robot learning an assembly task from a human video. In these real-world scenarios, the reference video is filmed from a third-person perspective (exocentric), while the person or robot executing the task experiences it from a first-person perspective (egocentric).
Detecting mistakes in this setup—termed Ego$\rightarrow$Exo Imitation Error Detection—presents three major challenges:
- Temporal Misalignment: The demonstration and imitation videos are captured asynchronously, meaning they differ in timing, pace, and total duration.
- Heavy Redundancy: Long, untrimmed videos contain substantial non-informative background frames. Surprisingly, standard models actually perform worse as the number of input frames increases because these redundant frames distract the attention mechanism.
- Pronounced View-Domain Gap: Egocentric videos focus heavily on local hand-object interactions, whereas exocentric videos capture global body posture and scene layout. Their appearance and motion statistics look completely different.
Prior error-detection methods operate strictly within a single viewpoint. SAVA-X is designed to bridge this unaligned, cross-view gap.
Methods
SAVA-X tackles these challenges using a unified Align–Fuse–Detect framework consisting of three novel components:
- Gated Adaptive Sampling (AS): To eliminate distracting redundancy, the model uses a Gumbel Top-K straight-through estimator to select only the most salient frames. It pairs this discrete selection with a differentiable "residual gating" path, ensuring stable gradient flow during training.
- Scene-Adaptive View Embeddings (SVE): Instead of using static view tokens (like standard positional embeddings), SAVA-X maintains a learnable, shared view-scene dictionary. It dynamically generates view embeddings based on the scene context, effectively narrowing the domain gap between the first- and third-person perspectives.
- Bidirectional Cross-Attention Fusion (BiX): Rather than simply concatenating the video features, SAVA-X applies symmetric cross-attention. The exocentric stream provides global structural and step-ordering cues to the egocentric stream, while the egocentric stream passes fine-grained hand-object details back to the exocentric stream.
Finally, the fused representations are passed to a 1D multi-scale Deformable DETR encoder-decoder to output precise temporal spans and predict whether each step was executed correctly or contained an error (such as an omission, sequence disorder, or execution deviation).
Datasets
The model was trained and evaluated on EgoMe, currently the only large-scale dataset that provides paired, asynchronous egocentric and exocentric videos complete with fine-grained procedural step annotations and error labels. The dataset consists of 7,902 video pairs (approximately 82.8 hours of footage), split into standard train, validation, and test sets.
Key Insights and Ideas
- More Data is a Double-Edged Sword: Without smart filtering, feeding higher frame-rate videos into standard models degrades error-detection performance. SAVA-X's adaptive sampling proves that sparsifying the timeline to key moments is essential for stabilizing cross-view alignment.
- Asymmetric Attention Roles: Ablation studies revealed that the Exo$\rightarrow$Ego attention direction is significantly more critical than Ego$\rightarrow$Exo. This makes intuitive sense: global demonstration cues are vital for establishing the correct step boundaries and ordering on the user's timeline.
- Interleaving the Feature Space: Visualizing the feature distributions using t-SNE showed that before applying SVE, ego and exo features form completely separate clusters. Post-SVE, the distributions become highly interleaved, proving the scene-adaptive dictionary successfully enforces view-invariance.
Key Results
SAVA-X was compared against strong baselines adapted from Dense Video Captioning (DVC) and Temporal Action Localization (TAL). It consistently achieved state-of-the-art performance across both validation and test sets:
- Error Detection Accuracy: On the EgoMe validation set, SAVA-X achieved a Mean AUPRC (Area Under the Precision-Recall Curve) of 22.36, marking a +2.67 absolute (+13.56% relative) improvement over the strongest baseline (Exo2EgoDVC).
- Localization Quality: SAVA-X also improved step boundary localization, yielding a mean temporal Intersection-over-Union (tIoU) of 59.31.
- Single-View Baseline Drop: Running the baseline model on egocentric data alone caused performance to plummet (Mean AUPRC dropped to 12.79), confirming that cross-view reference data is absolutely essential for accurate procedural verification.
Limitations and Future Work
While SAVA-X demonstrates robust performance, the authors note some areas for improvement:
- Decisive vs. Complementary Flows: The Ego$\rightarrow$Exo information flow was found to be helpful but not entirely decisive. Future work could optimize how fine-grained egocentric details are leveraged to verify exocentric alignment.
- Frozen Backbones: The framework currently relies on a frozen feature extractor (TSP). Exploring end-to-end training with larger, modern multimodal video foundation models could unlock even higher accuracy, albeit at a steeper computational cost.
Summary updated