SAVA-X: Ego-to-Exo Imitation Error Detection via Scene-Adaptive View Alignment and Bidirectional Cross View Fusion

Xiang Li, Heqian Qiu, Lanxiao Wang, Benliu Qiu, Fanman Meng, Linfeng Xu, Hongliang Li

TL;DR

SAVA-X introduces a novel framework for Ego$\rightarrow$Exo Imitation Error Detection, identifying procedural mistakes when a first-person (egocentric) execution attempts to copy a third-person (exocentric) demonstration. By combining adaptive frame sampling, scene-aware view embeddings, and bidirectional cross-view fusion, the model successfully overcomes massive temporal misalignments and domain shifts to accurately localize and flag execution errors.

What Problem Does This Paper Address?

Imagine a student nurse trying to replicate a complex medical protocol after watching an instructor's demonstration, or a factory robot learning an assembly task from a human video. In these real-world scenarios, the reference video is filmed from a third-person perspective (exocentric), while the person or robot executing the task experiences it from a first-person perspective (egocentric).

Detecting mistakes in this setup—termed Ego$\rightarrow$Exo Imitation Error Detection—presents three major challenges:

  1. Temporal Misalignment: The demonstration and imitation videos are captured asynchronously, meaning they differ in timing, pace, and total duration.
  2. Heavy Redundancy: Long, untrimmed videos contain substantial non-informative background frames. Surprisingly, standard models actually perform worse as the number of input frames increases because these redundant frames distract the attention mechanism.
  3. Pronounced View-Domain Gap: Egocentric videos focus heavily on local hand-object interactions, whereas exocentric videos capture global body posture and scene layout. Their appearance and motion statistics look completely different.

Prior error-detection methods operate strictly within a single viewpoint. SAVA-X is designed to bridge this unaligned, cross-view gap.

Figure 1. Top: Schematic of the Ego→Exo imitation-error detection task. The system localizes steps on the ego time- line and judges each step by semantic adherence to the exo- centric demonstration, rather than rigid speed/pose matching. Bottom-left: Baseline exhibits a counterintuitive performance drop as the number of input frames increases, partly because redundant frames in the videos introduce distraction. Bottom: There is a pronounced domain shift between Ego and Exo, the distribution of similarities between video-level features of demonstration–imitation pairs is overly dispersed. Bottom- right: A key challenge is how to effectively fuse information from Ego and Exo videos to accomplish the task.

Methods

SAVA-X tackles these challenges using a unified Align–Fuse–Detect framework consisting of three novel components:

  1. Gated Adaptive Sampling (AS): To eliminate distracting redundancy, the model uses a Gumbel Top-K straight-through estimator to select only the most salient frames. It pairs this discrete selection with a differentiable "residual gating" path, ensuring stable gradient flow during training.
  2. Scene-Adaptive View Embeddings (SVE): Instead of using static view tokens (like standard positional embeddings), SAVA-X maintains a learnable, shared view-scene dictionary. It dynamically generates view embeddings based on the scene context, effectively narrowing the domain gap between the first- and third-person perspectives.
  3. Bidirectional Cross-Attention Fusion (BiX): Rather than simply concatenating the video features, SAVA-X applies symmetric cross-attention. The exocentric stream provides global structural and step-ordering cues to the egocentric stream, while the egocentric stream passes fine-grained hand-object details back to the exocentric stream.

Finally, the fused representations are passed to a 1D multi-scale Deformable DETR encoder-decoder to output precise temporal spans and predict whether each step was executed correctly or contained an error (such as an omission, sequence disorder, or execution deviation).

Figure 2. Overview of SAVA-X. (1) A frozen video encoder extracts per-frame features from the exocentric demonstration and egocentric imitation streams. We apply gated adaptive sampling (Sec. 3.2)—hard Top-K with residual gating, using self-attention scoring for Exo and Exo-conditioned cross-attention scoring for Ego to select key segments. (2) We inject scene-aware dictionary view embeddings (Sec. 3.3) together with temporal positions (multi-level), regularized by attention-entropy and prototype-diversity terms, to mitigate cross-view domain shifts. (3) We perform bidirectional cross-attention fusion (Sec. 3.4) with learnable gating to align and aggregate complementary cues, yielding a fused sequence that a deformable Transformer encoder–decoder converts into first-person temporal spans and imitation-correctness predictions. Training uses Hungarian set prediction with LDVC and LImit.

Datasets

The model was trained and evaluated on EgoMe, currently the only large-scale dataset that provides paired, asynchronous egocentric and exocentric videos complete with fine-grained procedural step annotations and error labels. The dataset consists of 7,902 video pairs (approximately 82.8 hours of footage), split into standard train, validation, and test sets.

Key Insights and Ideas

  • More Data is a Double-Edged Sword: Without smart filtering, feeding higher frame-rate videos into standard models degrades error-detection performance. SAVA-X's adaptive sampling proves that sparsifying the timeline to key moments is essential for stabilizing cross-view alignment.
  • Asymmetric Attention Roles: Ablation studies revealed that the Exo$\rightarrow$Ego attention direction is significantly more critical than Ego$\rightarrow$Exo. This makes intuitive sense: global demonstration cues are vital for establishing the correct step boundaries and ordering on the user's timeline.
  • Interleaving the Feature Space: Visualizing the feature distributions using t-SNE showed that before applying SVE, ego and exo features form completely separate clusters. Post-SVE, the distributions become highly interleaved, proving the scene-adaptive dictionary successfully enforces view-invariance.

Key Results

SAVA-X was compared against strong baselines adapted from Dense Video Captioning (DVC) and Temporal Action Localization (TAL). It consistently achieved state-of-the-art performance across both validation and test sets:

  • Error Detection Accuracy: On the EgoMe validation set, SAVA-X achieved a Mean AUPRC (Area Under the Precision-Recall Curve) of 22.36, marking a +2.67 absolute (+13.56% relative) improvement over the strongest baseline (Exo2EgoDVC).
  • Localization Quality: SAVA-X also improved step boundary localization, yielding a mean temporal Intersection-over-Union (tIoU) of 59.31.
  • Single-View Baseline Drop: Running the baseline model on egocentric data alone caused performance to plummet (Mean AUPRC dropped to 12.79), confirming that cross-view reference data is absolutely essential for accurate procedural verification.

Limitations and Future Work

While SAVA-X demonstrates robust performance, the authors note some areas for improvement:

  • Decisive vs. Complementary Flows: The Ego$\rightarrow$Exo information flow was found to be helpful but not entirely decisive. Future work could optimize how fine-grained egocentric details are leveraged to verify exocentric alignment.
  • Frozen Backbones: The framework currently relies on a frozen feature extractor (TSP). Exploring end-to-end training with larger, modern multimodal video foundation models could unlock even higher accuracy, albeit at a steeper computational cost.

Summary updated

Abstract

Error detection is crucial in industrial training, healthcare, and assembly quality control. Most existing work assumes a single-view setting and cannot handle the practical case where a third-person (exo) demonstration is used to assess a first-person (ego) imitation. We formalize Ego$\rightarrow$Exo Imitation Error Detection: given asynchronous, length-mismatched ego and exo videos, the model must localize procedural steps on the ego timeline and decide whether each is erroneous. This setting introduces cross-view domain shift, temporal misalignment, and heavy redundancy. Under a unified protocol, we adapt strong baselines from dense video captioning and temporal action detection and show that they struggle in this cross-view regime. We then propose SAVA-X, an Align-Fuse-Detect framework with (i) view-conditioned adaptive sampling, (ii) scene-adaptive view embeddings, and (iii) bidirectional cross-attention fusion. On the EgoMe benchmark, SAVA-X consistently improves AUPRC and mean tIoU over all baselines, and ablations confirm the complementary benefits of its components. Code is available at https://github.com/jack1ee/SAVAX.
BibTeX
@inproceedings{li2026sava,
  author = {Xiang Li and Heqian Qiu and Lanxiao Wang and Benliu Qiu and Fanman Meng and Linfeng Xu and Hongliang Li},
  title = {{SAVA-X: Ego-to-Exo Imitation Error Detection via Scene-Adaptive View Alignment and Bidirectional Cross View Fusion}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=ycHcOdhmfH},
}
Figure