DroNeRF: Real-Time Multi-Agent Drone Pose Optimization for Computing Neural Radiance Fields

Dipam Patel, Phu Pham, Aniket Bera

TL;DR

DroNeRF is an optimization framework that allows a swarm of low-cost drones to autonomously position themselves for high-quality 3D reconstruction using Neural Radiance Fields (NeRF). By prioritizing an object's specific geometry rather than flying in a standard circle, the system achieves over 100% improvement in reconstruction accuracy using only a handful of images.

What Problem Does This Paper Address?

Neural Radiance Fields (NeRF) have revolutionized 3D reconstruction, but they are notoriously sensitive to camera placement. If you have a limited number of images (a "sparse" set), where you take those photos matters immensely.

Most current methods rely on a "hemisphere" approach—placing cameras at uniform intervals around an object. While this works for a perfect sphere, it fails for real-world objects that are non-symmetric or have concavities (like a person sitting at a desk or a guitar). Furthermore, high-quality 3D scanning usually requires expensive LiDAR or RGB-D sensors. DroNeRF aims to solve this using a swarm of cheap, monocular (standard) camera drones that "smartly" decide where to fly to capture the most important geometric details.

Methods

The DroNeRF pipeline consists of several clever stages to move from "blind flight" to "optimized capture":

  1. Seed Geometry Computation: To understand what they are looking at, four drones initially position themselves at 90-degree intervals. They use a Laplacian of Gaussian (LoG) filter to detect "regions of interest" rather than just edges. This allows the system to build a crude 3D bounding box of the target.
  2. Pose Optimization: Using the crude geometry, the system runs a Simulated Annealing (SA) algorithm. This optimization technique iteratively searches for drone positions that maximize "mesh visibility"—essentially finding the angles where the most complex parts of the object are visible.
  3. Closed-Loop Control: The drones don't use external GPS or motion capture. Instead, they use a PID controller that treats the object's centroid as the setpoint. It adjusts the drone's Roll, Pitch, Yaw, and Throttle in real-time to reach the optimized coordinates.
  4. Fast Reconstruction: Once the optimized images (typically only 20-24 total) are captured, they are fed into Instant-NGP, a high-speed NeRF implementation that allows for near real-time 3D rendering.

Datasets

The researchers evaluated their system using real-world captures rather than static datasets, as the core contribution is the active positioning of the drones. They tested four distinct scenes:

  • Human: A person sitting at a table (complex, non-symmetric).
  • Guitar: An object with thin structures and specific orientations.
  • Robot Dog: A complex toy with various concavities.
  • Recycle Bin: A more standard geometric object.

Key Insights and Ideas

  • Geometry over Symmetry: The "aha moment" here is that uniform camera placement is inefficient. By moving a drone higher or closer to a specific concavity (like the space between a person's arms and a table), the NeRF model gains significantly more information than it would from a standard circular path.
  • Self-Localization via Object: By using the detected object's bounding box to calculate errors for the PID controller, the drones can maintain precise positioning without needing expensive external localization systems like Vicon.
  • Sparse is Enough: The paper proves that where you take the photo is more important than how many photos you take. They achieved high-fidelity results with just 24 images, whereas standard NeRF captures often require hundreds.

Key Results

The quantitative improvements over the "standard" uniform drone placement were dramatic:

  • PSNR (Peak Signal-to-Noise Ratio): In the "Robot Dog" scene, the optimized placement resulted in a 123% increase in PSNR (from 10.18 to 22.72).
  • SSIM (Structural Similarity): Across all scenes, structural accuracy improved by at least 53%.
  • Qualitative Quality: Visual results show that the optimized flight paths eliminate much of the "blur" and "floaters" typically seen in sparse-view NeRFs, especially around the most complex parts of the subject.

Limitations and Future Work

The authors acknowledge a few hurdles for the current version of DroNeRF:

  • Static Scenes Only: The algorithm assumes the object is stationary. If the subject moves during the multi-drone capture, the resulting 3D model becomes noisy or fails.
  • Initial Seed Reliance: The optimization depends on the accuracy of the initial "crude" bounding box. If the first four images are poor, the subsequent optimization may struggle.
  • Future Directions: The team aims to expand this to dynamic scenes (moving objects) and improve the drones' ability to autonomously find the subject of interest in a cluttered environment before beginning the optimization.

Summary updated

Abstract

We present a novel optimization algorithm called DroNeRF for the autonomous positioning of monocular camera drones around an object for real-time 3D reconstruction using only a few images. Neural Radiance Fields or NeRF, is a novel view synthesis technique used to generate new views of an object or scene from a set of input images. Using drones in conjunction with NeRF provides a unique and dynamic way to generate novel views of a scene, especially with limited scene capabilities of restricted movements. Our approach focuses on calculating optimized pose for individual drones while solely depending on the object geometry without using any external localization system. The unique camera positioning during the data-capturing phase significantly impacts the quality of the 3D model. To evaluate the quality of our generated novel views, we compute different perceptual metrics like the Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure(SSIM). Our work demonstrates the benefit of using an optimal placement of various drones with limited mobility to generate perceptually better results.
BibTeX
@inproceedings{patel2023dronerf,
  author = {Dipam Patel and Phu Pham and Aniket Bera},
  title = {{DroNeRF: Real-Time Multi-Agent Drone Pose Optimization for Computing Neural Radiance Fields}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2023},
  doi = {10.1109/iros55552.2023.10342420},
}
Figure