Video Synchronization Using Temporal Signals from Epipolar Lines

Dmitry Pundik, Yael MosesECCV 2010 · DOI

TL;DR

This paper introduces a novel method for synchronizing video streams from multiple static cameras by analyzing temporal signals derived from epipolar lines. This approach bypasses the need for complex feature matching or 3D reconstruction, making it robust to significant scene motion and occlusions, and enabling real-time synchronization even with frame drops.

What Problem Does This Paper Address?

In multi-camera systems, such as those used for surveillance or filmmaking, precise time synchronization between video streams is crucial for effective analysis. However, technical imperfections, particularly frame dropping, can cause synchronization to degrade over time. Traditional methods for maintaining synchronization often rely on expensive hardware (like GPS), complex algorithms that struggle with dynamic scenes, or assumptions about camera setups that don't always hold. This paper addresses the challenge of robustly and continuously synchronizing video streams from static cameras, even in the presence of substantial scene motion and without requiring prior knowledge of camera calibration or feature correspondence.

Methods

The core of the proposed method lies in defining and matching "temporal signals" derived from epipolar lines.

  • Epipolar Geometry: The method assumes a known fundamental matrix between camera pairs, which defines epipolar lines. These lines provide a spatial correspondence constraint without requiring pixel-level matching.
  • Temporal Signal Definition: For each epipolar line, a temporal signal is computed. This signal is an integration of motion indicators (derived from background subtraction) along the epipolar line over a short time interval (at least 2 seconds). This approach avoids the problem of changing pixel correspondences due to 3D motion.
  • Probabilistic Registration: The synchronization is achieved by matching these temporal signals from corresponding epipolar lines across the two video sequences. A probabilistic framework, specifically a maximum a posteriori estimation, is used to find the optimal time shift. This involves calculating the likelihood of observing the temporal signals given a particular time shift.
  • Online and Robustness: The method is designed for an online framework, processing data in short time intervals. It also incorporates a filtering step to remove epipolar lines with negligible motion, improving robustness. The framework can detect and correct synchronization errors, such as frame drops, in near real-time.

Datasets

The paper evaluates its method on three sets of video sequences:

  • Set 1: An indoor scenario with a dense crowd (approx. 30 people) walking about, with cameras at ~6 meters elevation and significant overlap. Videos were recorded at 25 fps, 640x480 resolution.
  • Set 2: Similar to Set 1 but with partially overlapping fields of view, representing a more challenging case due to different viewing angles. Total runtime of both sequences was 5000 frames (~3.33 minutes).
  • Set 3: A dark outdoor scene with limited motion, recorded at 15 fps, 640x512 resolution.

The paper also mentions testing on a pair of short videos from literature (a single car in a parking lot) to verify its performance on different sequences.

Key Insights and Ideas

  • Epipolar Lines as a Synchronization Primitive: The paper demonstrates that epipolar lines, derived from fundamental matrices, can serve as a robust basis for temporal signal extraction and matching, bypassing the need for complex point or feature correspondences.
  • Temporal Signals on Lines vs. Pixels: Defining temporal signals along entire epipolar lines, rather than at individual pixels, effectively handles the ambiguity introduced by 3D motion and occlusions, which constantly change pixel correspondences.
  • Online Synchronization with Frame Drop Resilience: The method's ability to perform synchronization in an online fashion and its demonstrated resilience to frame drops (correcting shifts within seconds) is a significant practical advantage for real-world systems.
  • Calibration Verification Potential: The method's sensitivity to the epipolar geometry suggests it could be used as a tool to detect calibration failures, acting as a first step towards recalibration.

Key Results

  • The method achieves synchronization with a correct time shift for all tested sets, with the correct time shift being $\Delta t = -1$ frame for all.
  • For Set 1, approximately 50% of results had high confidence, and 95% were correct, yielding a high-confidence result every 1.6 seconds on average.
  • Set 2 showed 100% correct results, but only 12% had high confidence, attributed to the smaller overlap between camera views.
  • Set 3 also achieved 100% correct results, with 13% having high confidence.
  • The method demonstrated robustness to frame drops, with a reaction time of approximately 7-8 seconds to detect and correct a shift.
  • The use of a prior on the time shift distribution (based on previous iterations) was shown to reduce instability in low-confidence results without impairing other results.

Limitations and Future Work

  • Failure Cases: The method may fail in rare cases where an object moves strictly along an epipolar line, leading to ambiguous temporal matching. Incorrect synchronization is also expected if an object moves across non-overlapping camera regions.
  • Calibration Dependency: The method relies on a correctly computed fundamental matrix. If the epipolar geometry is incorrect, synchronization fails. However, this sensitivity can be leveraged to detect calibration failures.
  • Future Work: The authors suggest extending the approach to handle systems with more than two cameras, which they believe can be naturally achieved due to the probabilistic nature of the algorithm. They also intend to study the extension to handle more complex scenarios.

Summary updated

BibTeX
@inproceedings{pundik2010video,
  author = {Dmitry Pundik and Yael Moses},
  title = {{Video Synchronization Using Temporal Signals from Epipolar Lines}},
  booktitle = {European Conference on Computer Vision},
  year = {2010},
  doi = {10.1007/978-3-642-15558-1\_2},
}
Figure