Sync-NeRF: Generalizing Dynamic NeRFs to Unsynchronized Videos

Seoha Kim, Jeongmin Bae, Youngsik Yun, Hahyun Lee, Gun Bang, Youngjung Uh

TL;DR

This paper introduces Sync-NeRF, a novel method that enables dynamic Neural Radiance Fields (NeRFs) to reconstruct scenes from unsynchronized multi-view videos. By learning per-camera time offsets, Sync-NeRF effectively synchronizes the input streams, significantly improving reconstruction quality and resolving artifacts that plague existing methods when faced with imperfect temporal alignment.

What Problem Does This Paper Address?

Reconstructing dynamic 3D scenes from multi-view videos has seen significant advancements with Neural Radiance Fields (NeRFs). However, a critical limitation arises when these videos are not perfectly synchronized. Existing dynamic NeRF methods typically assume that corresponding frames across different camera views capture the exact same moment in time. When this assumption is violated, due to factors like imperfect camera calibration or recording setups, the models struggle to accurately represent the scene's motion. This leads to severe artifacts, ghosting, and a failure to reconstruct even the training views correctly. While traditional synchronization methods exist (e.g., timecodes, audio peaks), they often require additional hardware, are susceptible to noise, or can be inaccurate, especially in complex dynamic scenarios. This paper tackles the problem of reconstructing dynamic scenes from these "real-world" unsynchronized multi-view videos.

Methods

The core innovation of Sync-NeRF lies in introducing learnable per-camera time offsets, denoted as $\delta_k$ for camera $k$. Instead of assuming a single global time $t$ for all views, Sync-NeRF models the time input for camera $k$ as $t_k = t + \delta_k$. These offsets are jointly optimized with the NeRF parameters using a standard reconstruction loss (MSE between rendered and ground truth pixels). This approach effectively "shifts" the temporal axis of each video to align it with others, without requiring explicit synchronization beforehand.

Sync-NeRF is designed to be versatile and can be applied to different types of dynamic NeRF architectures:

  • Implicit Function-Based Approaches: For methods that use per-frame latent embeddings (like MixVoxels), Sync-NeRF modifies the time input to the implicit neural representation that generates these embeddings. This allows the model to produce temporal embeddings corresponding to the calibrated time $t_k$.
  • Grid-Based Models: For methods that rely on feature grids (like K-Planes), Sync-NeRF adjusts the time coordinate used to query the grid. Bilinear interpolation on the grid then naturally handles the continuous temporal embedding at the calibrated time $t_k$.

The paper also highlights that optimizing these time offsets can be done with a test-time optimization approach, allowing Sync-NeRF to adapt to unsynchronized test views even if the training data was synchronized.

Datasets

The authors evaluate Sync-NeRF on two datasets:

  • Plenoptic Video Dataset: A widely used dataset for 4D scene reconstruction. The authors intentionally unsynchronize this dataset by randomly shifting temporal frames with a normal distribution (mean 0, std dev 5, rounded to integers) to simulate real-world unsynchronization.
  • Unsynchronized Dynamic Blender Dataset: A newly created synthetic dataset. This dataset is generated by rendering free Blender assets (box, fox, deer) with motion, then applying random time offsets to the rendered videos. This synthetic approach provides ground truth time offsets, allowing for precise evaluation of the learned offsets.

Key Insights and Ideas

  • The "Synchronization Gap" is a Major Bottleneck: The paper convincingly demonstrates that even state-of-the-art dynamic NeRFs fail dramatically when faced with unsynchronized inputs, highlighting that temporal alignment is as crucial as spatial and appearance modeling.
  • Time Offsets as Implicit Synchronization: Learning per-camera time offsets is a remarkably simple yet effective way to achieve synchronization. It bypasses the need for external synchronization hardware or complex audio/visual analysis.
  • Versatility Across Architectures: Sync-NeRF is not tied to a specific dynamic NeRF backbone. Its ability to integrate seamlessly with both implicit function-based and grid-based models underscores its general applicability.
  • Ground Truth Offsets are Rare: The creation of the Unsynchronized Dynamic Blender Dataset is crucial for quantitatively evaluating the accuracy of the learned time offsets, as real-world datasets often lack perfect synchronization and thus ground truth offsets.

Key Results

Sync-NeRF consistently and significantly improves the performance of baseline dynamic NeRF models (MixVoxels and K-Planes) on unsynchronized datasets.

  • Rendering Quality: Quantitative metrics (PSNR, SSIM, LPIPS) show substantial improvements across the board. For instance, on the Unsynchronized Plenoptic Video Dataset, Sync-MixVoxels improves PSNR by ~0.57 dB and Sync-K-Planes-H by ~1.28 dB over their unsynchronized counterparts. On the Unsynchronized Dynamic Blender Dataset, Sync-MixVoxels achieves PSNR gains of over 6 dB.
  • Artifact Reduction: Qualitative results (rendered images, spatiotemporal images) clearly show the elimination of artifacts and ghosting, leading to much sharper and more accurate reconstructions of dynamic elements like hands and objects.
  • Time Offset Accuracy: The learned time offsets are highly accurate, with a Mean Absolute Error (MAE) of approximately 0.01 seconds on the synthetic dataset, which is less than one-third of a frame.
  • Robustness: The method demonstrates robustness to varying lengths of unsynchronization (1.5x, 2x) and even provides improvements on datasets assumed to be synchronized, suggesting that even small temporal gaps can be detrimental.

Limitations and Future Work

  • Causal Handheld Cameras: The authors acknowledge that their method does not yet generalize to causal handheld camera scenarios, which often involve more complex motion and potential occlusions. This is suggested as an interesting avenue for future research.
  • Computational Cost for Implicit Functions: While grid-based models experience negligible overhead, implicit function-based temporal embeddings might require additional training time and memory. However, the authors note that these embeddings can be pre-computed during inference, mitigating the overhead.
  • Generalization to More Complex Dynamics: While Sync-NeRF handles dynamic scenes well, extending it to even more complex and unpredictable dynamic phenomena could be a future direction.

Summary updated

Abstract

Recent advancements in 4D scene reconstruction using neural radiance fields (NeRF) have demonstrated the ability to represent dynamic scenes from multi-view videos. However, they fail to reconstruct the dynamic scenes and struggle to fit even the training views in unsynchronized settings. It happens because they employ a single latent embedding for a frame while the multi-view images at the same frame were actually captured at different moments. To address this limitation, we introduce time offsets for individual unsynchronized videos and jointly optimize the offsets with NeRF. By design, our method is applicable for various baselines and improves them with large margins. Furthermore, finding the offsets naturally works as synchronizing the videos without manual effort. Experiments are conducted on the common Plenoptic Video Dataset and a newly built Unsynchronized Dynamic Blender Dataset to verify the performance of our method. Project page: https://seoha-kim.github.io/sync-nerf
BibTeX
@inproceedings{kim2024sync,
  author = {Seoha Kim and Jeongmin Bae and Youngsik Yun and Hahyun Lee and Gun Bang and Youngjung Uh},
  title = {{Sync-NeRF: Generalizing Dynamic NeRFs to Unsynchronized Videos}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i3.28057},
}
Figure