3D Motion Magnification: Visualizing Subtle Motions from Time-Varying Radiance Fields

Brandon Y. Feng, Hadi Alzayer, Michael Rubinstein, William T. Freeman, Jia-bin HuangICCV 2023

TL;DR

This paper introduces 3D motion magnification, enabling the visualization of subtle, imperceptible motions in dynamic 3D scenes captured by moving cameras. By leveraging neural radiance fields (NeRF) and adapting Eulerian motion magnification principles to the feature embedding space, the method achieves robust motion amplification from novel viewpoints, even in challenging real-world scenarios.

What Problem Does This Paper Address?

Visualizing subtle motions, like vibrations or slight deformations, is crucial for various applications, from structural health monitoring to understanding biological processes. While 2D motion magnification techniques have been successful for videos captured with static cameras, they fail dramatically when the camera itself is moving. This is because these methods cannot disentangle the camera's motion from the scene's actual subtle movements, leading to severe artifacts. The core problem this paper addresses is extending motion magnification to dynamic 3D scenes captured by moving cameras, allowing for visualization from novel viewpoints and separating camera motion from object motion.

Methods

The paper's core innovation lies in adapting Eulerian motion magnification principles to the domain of neural radiance fields (NeRF). Instead of directly magnifying color variations in 2D images, they propose magnifying variations within the feature embeddings of 3D points.

Here's a breakdown of their approach:

  • 3D Scene Representation with NeRF: They utilize NeRF, which represents a 3D scene as a continuous function that maps a 3D point and viewing direction to color and opacity. This function typically involves a coordinate transformation (mapping 3D points to embeddings) and a projection function (mapping embeddings to color/opacity).
  • Time-Varying Embeddings: To capture subtle motions, they allow the embedding function (E) to vary over time, while keeping the projection function (MLP) fixed. This means that for a given 3D point, its representation changes subtly over time to reflect the scene's dynamics.
  • Eulerian Motion Magnification in Embedding Space: They adapt Eulerian motion magnification techniques to these time-varying embeddings. This involves:
  • Temporal Filtering: Analyzing the embeddings of a fixed 3D point across different timesteps.
  • Frequency Isolation: Using Fourier transforms and bandpass filters to isolate variations within a specific frequency range corresponding to the desired motion.
  • Amplification: Amplifying these temporal variations in the embedding space.
  • Magnified Rendering: Feeding these amplified embeddings back into the NeRF projection function to render motion-magnified novel views.
  • Two Embedding Strategies:
    • Positional Encoding: For standard NeRFs using analytical positional encoding, they either shift the 3D point's coordinates or shift the encoding frequencies themselves.
  • Tri-plane Embedding (Preferred): For learned tri-plane representations (which are more efficient), they train a separate tri-plane for each timestep. This allows them to directly apply 2D motion magnification techniques (linear or phase-based) to each feature plane of the tri-plane representation. This approach is found to be more robust and effective.

Datasets

  • Synthetic Scenes: They generated synthetic scenes using Blender, simulating periodic vibrations in object parts. These scenes were rendered at 30 fps for one second, with simulated motions ranging from 3 Hz to 5 Hz. Ground truth magnified motions were also generated for quantitative evaluation.
  • Real-world Multi-view Video: They used the HumanNeRF dataset [66], which provides short videos from six cameras simultaneously capturing a scene with a person.
  • Real-world Single-camera Video: For single-camera scenarios, they designed a two-stage capture procedure: first, a static scene with a moving camera to train a static NeRF, and second, a dynamic scene with the same moving camera to finetune the embedding function. They showcase results on scenes with a gymnast, a person balancing, and a person breathing.

Key Insights and Ideas

  • Eulerian Principle Extends to Embeddings: The most significant insight is that the Eulerian principle for motion magnification, which works by analyzing temporal variations at fixed locations, successfully generalizes from color space to the feature embedding space of neural radiance fields. This opens up new avenues for applying classical signal processing techniques to implicit neural representations.
  • Separating Camera Motion from Scene Motion: By operating on the learned embeddings, their method can naturally disentangle the camera's motion from the subtle motions of the scene's objects, a critical limitation of prior 2D methods.
  • Tri-plane Embeddings are Particularly Well-Suited: The tri-plane representation, by decomposing the 3D scene into 2D feature planes, provides a natural bridge to existing 2D video magnification techniques, leading to superior performance and robustness, especially with the phase-based approach.
  • Magnifying Subtle 3D Motions is Possible: The paper demonstrates that even very small, imperceptible motions in 3D can be effectively captured and amplified using this NeRF-based approach, revealing details not visible to the naked eye.

Key Results

  • Outperforms 2D Methods: Their 3D motion magnification methods significantly outperform traditional 2D video magnification techniques on both synthetic and real-world data, especially when dealing with camera motion.
  • Robustness to Noise and Viewpoint Changes: The phase-based magnification on tri-plane embeddings shows greater robustness to noise in training views and maintains quality even when analyzing views deviating from the original capture angles.
  • Successful Real-World Demonstrations: They successfully demonstrate motion magnification on real-world multi-view and single-camera setups, including challenging scenarios like handheld captures with camera shake.
  • Quantitative Improvements: Figures 5, 8, and 9 show quantitative comparisons using SSIM and LPIPS metrics, demonstrating superior performance of their proposed methods (especially Phase-Tri-Plane) across various factors, angles, and noise levels.

Limitations and Future Work

  • Data Quality and Pose Estimation: The performance of NeRF, and consequently their method, is highly dependent on accurate camera pose estimation. Inaccurate poses can lead to ambiguity between camera and scene motion, hindering magnification. Real-world data needs to be captured under conditions where accurate camera intrinsics and extrinsics are accessible.
  • Blurriness and Camera Shake: Blurriness from defocus and camera shake in real-world captures can degrade NeRF quality and thus the magnification results.
  • Future Work: The authors suggest that their work motivates further research into integrating traditional signal processing techniques with neural fields. Future work could explore more advanced NeRF architectures and magnification strategies for even more complex dynamic scenes.

Summary updated

BibTeX
@inproceedings{feng2023d,
  author = {Brandon Y. Feng and Hadi Alzayer and Michael Rubinstein and William T. Freeman and Jia-bin Huang},
  title = {{3D Motion Magnification: Visualizing Subtle Motions from Time-Varying Radiance Fields}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
}
Figure