BAD-Gaussians: Bundle Adjusted Deblur Gaussian Splatting

Lingzhe Zhao, Peng Wang, Peidong Liu

Introduction

Recent advancements in 3D reconstruction, particularly Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3D-GS), have revolutionized our ability to synthesize novel views from sets of static images. However, these methods typically operate under the assumption of ideal input data: perfectly sharp images and highly accurate camera poses, often derived from Structure-from-Motion (SfM) tools like COLMAP. In real-world scenarios—such as handheld photography or robotics in low-light environments—motion blur is a frequent and destructive artifact.

BAD-Gaussians (Bundle Adjusted Deblur Gaussian Splatting) is the first motion deblurring framework built on the 3D-GS architecture. By jointly optimizing 3D scene parameters and camera motion trajectories, it recovers high-quality sharp scenes from severely blurred inputs, achieving real-time rendering speeds that significantly outperform previous implicit neural representations.

The Challenge of Motion Blur

Motion blur introduces a "triple threat" to standard 3D reconstruction pipelines:

  1. Supervision Failure: Blurred images provide ambiguous, "fuzzy" targets for the loss function, preventing the model from learning high-frequency details.
  2. Pose Failure: Feature-based matching in COLMAP often fails on blurry frames, leading to inaccurate or missing camera poses.
  3. Initialization Failure: 3D-GS relies heavily on a high-quality initial point cloud. If the initial SfM step fails due to blur, the subsequent reconstruction is often unrecoverable.

While previous "Deblur-NeRF" methods attempted to address these issues using implicit Multi-Layer Perceptrons (MLPs), they are computationally expensive, often requiring over 10 hours for training and rendering at sub-real-time speeds (<1 FPS).

The BAD-Gaussians Framework

BAD-Gaussians addresses these challenges by modeling the physical image formation process during the camera's exposure time. Instead of treating a camera pose as a single point in time, the framework treats it as a continuous trajectory.

Physical Blur Modeling

The system assumes that a blurred image is the integral (average) of several "virtual" sharp images captured along a short camera trajectory during the shutter interval. By using the 3D-GS framework, the authors can project Gaussians and average them across this trajectory efficiently, keeping the entire pipeline end-to-end differentiable.

Trajectory Learning and Joint Optimization

The camera's path during exposure is modeled using continuous splines. For synthetic data with constant velocity, linear interpolation is often sufficient. However, for real-world handheld footage, the framework utilizes Cubic B-splines to capture the complex nuances of human hand jitter.

During training, the model performs a form of "Bundle Adjustment." It uses gradients from the photometric loss to simultaneously refine the 3D Gaussians (representing the scene geometry and appearance) and the camera trajectories (representing the poses). This joint optimization allows the model to "pull" the geometry and poses into alignment even when starting from a sparse and noisy initial point cloud.

Experimental Evaluation

The authors evaluated BAD-Gaussians on several benchmarks, including the Deblur-NeRF (Synthetic and Real), BAD-NeRF Synthetic, and MBA-VO datasets.

Key Results

  • Efficiency: BAD-Gaussians achieves rendering speeds of >200 FPS, whereas previous state-of-the-art methods like BAD-NeRF operate at less than 1 FPS. Training is similarly accelerated, taking roughly 30 minutes compared to the 10+ hours required for NeRF-based alternatives.
  • Reconstruction Quality: On the Deblur-NeRF synthetic dataset, BAD-Gaussians outperformed the previous best methods by an average of 3.6 dB in PSNR. The explicit nature of 3D Gaussians allows for the recovery of much finer details compared to the smoothing effects often seen in MLP-based deblurring.
  • Pose Accuracy: The method significantly reduces Absolute Trajectory Error (ATE) compared to the initial poses provided by COLMAP, demonstrating that the framework effectively learns the true camera motion.

Discussion and Limitations

While BAD-Gaussians represents a significant leap forward, some limitations remain. The authors noted that the framework performs slightly worse than NeRF on specific sequences involving the infinite "sky," as 3D-GS inherently struggles with representing infinite depth compared to volumetric approaches. Additionally, the current model assumes a static scene where only the camera moves; handling dynamic objects within a blurry scene remains an open area for future research.

Conclusion

BAD-Gaussians demonstrates that explicit representations like 3D Gaussian Splatting are not only faster but also more effective at recovering intricate details from degraded inputs than implicit models. By integrating physical blur modeling and trajectory optimization, BAD-Gaussians provides a robust solution for high-quality 3D reconstruction in the presence of significant motion blur and pose uncertainty.

Summary updated

BibTeX
@inproceedings{zhao2024bad,
  author = {Lingzhe Zhao and Peng Wang and Peidong Liu},
  title = {{BAD-Gaussians: Bundle Adjusted Deblur Gaussian Splatting}},
  booktitle = {European Conference on Computer Vision},
  year = {2024},
  doi = {10.1007/978-3-031-72698-9\_14},
}
Figure