EgoSync: Self-Supervised Egocentric Video Synchronization through Mutual Temporal Embedding

Chunhua Shen, Mei-Ling Chen, Ayaan Verma, Karen Lindqvist, Soo-Jin ParkResearch Square 2026 · DOI

TL;DR

EgoSync is a self-supervised framework designed to synchronize egocentric (first-person) videos of the same activity captured by different people. By learning viewpoint-invariant temporal embeddings, it overcomes the extreme motion and visual diversity of head-mounted cameras to achieve state-of-the-art alignment without requiring manual synchronization labels.

What Problem Does This Paper Address?

Egocentric video—captured from head-mounted cameras like GoPro or AR glasses—offers a unique, "hands-on" view of human activity. However, synchronizing these videos is a nightmare for computer vision. Imagine two students in a cooking class: their head movements are erratic, they work at different speeds, and their hands often block the camera's view.

Traditional video alignment methods usually rely on "third-person" views where the camera is static and the viewpoint is consistent. In the egocentric world, the visual content changes drastically even when two people are doing the exact same thing. Existing models struggle to distinguish between the action (cracking an egg) and the viewpoint (looking down at the bowl vs. looking at the counter). EgoSync aims to bridge this gap, enabling applications like multi-user skill assessment and collaborative AR guidance.

Methods

The authors propose a self-supervised framework that maps video frames into a shared "temporal embedding space." The goal is for frames representing the same stage of an activity to have similar mathematical representations, regardless of who is wearing the camera.

The system relies on three core components:

  1. Mutual Temporal Embedding (MTE) Loss: Using a differentiable version of Dynamic Time Warping (Soft-DTW), the model finds the optimal non-linear path to align two video sequences. This accounts for the fact that one person might perform a task faster than another.
  2. Viewpoint-Invariant Contrastive (VIC) Loss: This is the "secret sauce." It forces the model to ignore viewpoint-specific details and focus on semantic activity. It treats frames from different videos that are temporally aligned as "positive pairs" and pushes them closer together in the embedding space, while pushing unrelated frames away.
  3. Temporal Smoothness Regularizer: To prevent the model from making erratic jumps, this regularizer encourages the "path" through the embedding space to progress linearly and smoothly over time.

Crucially, EgoSync is self-supervised. It doesn't need humans to tell it which frames match; it learns simply by being told that two videos depict the same general activity.

Datasets

The researchers evaluated EgoSync on three major egocentric benchmarks:

  • Ego4D: A massive dataset of daily life activities. They used 1,200 paired clips from social and procedural subsets.
  • EgoExo4D: A dataset featuring simultaneous "ego" (first-person) and "exo" (third-person) views. The authors extracted 850 ego-ego pairs for this task.
  • CMU-MMAC: A classic multimodal dataset containing cooking activities from different wearers.

Key Insights and Ideas

  • Disentanglement is Mandatory: You cannot align egocentric videos based on raw visual similarity because the backgrounds and angles are too different. The "aha moment" here is using the VIC loss to explicitly separate what is happening from where the camera is looking.
  • Alignment as a Pre-training Task: The paper shows that learning to synchronize videos actually makes the model better at other things. By forcing the model to understand the "flow" of an activity, it becomes significantly better at action recognition.
  • Soft-DTW + Contrastive Learning: While Soft-DTW is great for alignment, it often leads to "representation collapse" (where the model maps every frame to the same point). Combining it with contrastive learning provides the necessary "push" to keep the embeddings distinct and meaningful.

Key Results

EgoSync significantly outperformed existing baselines (like TCC, LAV, and CASA) across all datasets:

  • Accuracy: On Ego4D, it achieved 78.4% synchronization accuracy (within a 2-second tolerance), outperforming the previous best method by 6.1 percentage points.
  • Robustness: The gains were consistent across different time tolerances (1s, 2s, and 5s).
  • Downstream Impact: Using EgoSync-trained features improved "verb recognition" (identifying the action) by 3.2% on the Ego4D benchmark compared to standard pre-training.

Limitations and Future Work

While EgoSync is a major step forward, the authors note a few hurdles:

  • Activity Assumption: The model currently assumes it is given a pair of videos that already depict the same activity. In a real-world "wild" setting, you would first need an automated way to detect and pair these videos.
  • Scalability: The current framework is designed for pairs ($N=2$). Synchronizing a group of ten people simultaneously remains a non-trivial challenge.
  • Unstructured Data: The model excels at "procedural" tasks (like recipes or assembly) where there is a clear beginning, middle, and end. Applying this to less structured social interactions is a promising but difficult next step.

Summary updated

Abstract

<title>Abstract</title> Synchronizing egocentric videos captured by multiple wearers performing the same activity is a fundamental requirement for multi-view activity analysis, skill assessment, and collaborative AR guidance. Unlike third-person video synchronization, egocentric settings introduce severe viewpoint variations, rapid head motion, and object occlusions that make traditional alignment methods ineffective. We introduce EgoSync, a self-supervised framework for temporally synchronizing pairs of egocentric videos depicting the same procedural activity. Our approach learns a shared temporal embedding space through three complementary objectives: (1) a Mutual Temporal Embedding loss that aligns corresponding activity progress across viewpoints, (2) a Viewpoint-Invariant Contrastive loss that disentangles activity semantics from viewpoint-specific appearance, and (3) a Temporal Smoothness Regularizer that enforces locally consistent progression in the embedding space. EgoSync operates without manual synchronization annotations, using only the assumption that paired videos depict the same activity. Experiments on Ego4D, EgoExo4D, and CMU-MMAC demonstrate that EgoSync achieves 78.4% synchronization accuracy within a 2-second tolerance on Ego4D, outperforming adapted baselines by 6.1 percentage points. We further show that synchronized features improve downstream egocentric action recognition by 3.2% on Ego4D.
BibTeX
@inproceedings{shen2026egosync,
  author = {Chunhua Shen and Mei-Ling Chen and Ayaan Verma and Karen Lindqvist and Soo-Jin Park},
  title = {{EgoSync: Self-Supervised Egocentric Video Synchronization through Mutual Temporal Embedding}},
  booktitle = {Research Square},
  year = {2026},
  doi = {10.21203/rs.3.rs-8883832/v1},
}
Figure