DroNeRF: Real-time Multi-agent Drone Pose Optimization for Computing Neural Radiance Fields

Dipam Patel, Phu Pham, Aniket Bera

Abstract

Neural Radiance Fields (NeRF) have transformed 3D reconstruction, yet their performance remains highly dependent on camera placement, particularly in sparse-view scenarios. Traditional uniform sampling methods often fail to capture the intricate geometries of non-symmetric or concave objects. This paper introduces DroNeRF, an optimization framework that enables a swarm of low-cost drones to autonomously position themselves for high-quality 3D reconstruction. By prioritizing object-specific geometry over standard circular trajectories, DroNeRF achieves significant improvements in reconstruction accuracy—exceeding 100% in specific metrics—using only a handful of images.

1. Introduction

High-quality 3D scanning typically requires expensive hardware, such as LiDAR or RGB-D sensors, and extensive manual data collection. While NeRF provides a powerful alternative using standard RGB images, it is notoriously sensitive to the spatial distribution of the camera views. When limited to a "sparse" set of images, the placement of those cameras becomes the primary factor in reconstruction fidelity.

Most current autonomous capture methods utilize a "hemisphere" approach, placing cameras at uniform intervals around a target. While effective for simple, symmetric shapes, this approach is inefficient for real-world objects with complex concavities or thin structures. DroNeRF addresses this by utilizing a swarm of monocular camera drones that dynamically optimize their flight paths to capture the most informative geometric details of a target object.

2. Methodology

The DroNeRF pipeline facilitates a transition from initial discovery to optimized capture through a four-stage process:

2.1 Seed Geometry Computation

The process begins with four drones positioned at 90-degree intervals to establish a baseline. Rather than simple edge detection, the system employs a Laplacian of Gaussian (LoG) filter to identify "regions of interest." This allows the swarm to construct a crude 3D bounding box and initial geometric estimate of the target, providing the necessary context for further optimization.

2.2 Pose Optimization

Using the initial crude geometry, the system executes a Simulated Annealing (SA) algorithm. This optimization technique searches for drone positions that maximize "mesh visibility." By iteratively evaluating potential viewpoints, the algorithm identifies angles where the most complex or occluded parts of the object are visible, ensuring that the limited number of captures (typically 20-24 images) contains the highest possible information density.

2.3 Closed-Loop Control

To maintain precision without expensive external localization systems (like GPS or Vicon), DroNeRF utilizes a PID controller. The controller treats the object's detected centroid as the setpoint, adjusting the drone's Roll, Pitch, Yaw, and Throttle in real-time. This allows the drones to navigate to the optimized coordinates autonomously while maintaining the target in the center of the frame.

2.4 Fast Reconstruction

Once the optimized images are captured, they are processed using Instant-NGP, a high-speed NeRF implementation. This allows for near real-time 3D rendering, moving from raw image capture to a navigable 3D model in minutes.

3. Experimental Evaluation

The researchers evaluated DroNeRF using real-world captures of four distinct scenes, chosen for their geometric complexity:

  • Human: A person seated at a table, representing a complex, non-symmetric subject.
  • Guitar: An object featuring thin structures and specific orientation requirements.
  • Robot Dog: A toy with various concavities and intricate surfaces.
  • Recycle Bin: A control subject with more standard geometric properties.

3.1 Quantitative Results

The performance of DroNeRF was compared against standard uniform drone placement. The results demonstrated that where an image is taken is significantly more important than the total volume of images:

  • Peak Signal-to-Noise Ratio (PSNR): In the "Robot Dog" scene, optimized placement resulted in a 123% increase in PSNR (improving from 10.18 to 22.72).
  • Structural Similarity (SSIM): Across all tested scenes, structural accuracy improved by at least 53%.

3.2 Qualitative Observations

Visual analysis confirms that optimized flight paths effectively eliminate the "blur" and "floaters" (artifacts) typically associated with sparse-view NeRFs. The system specifically improved the clarity of complex areas, such as the space between a person's arms and a table, which are often lost in uniform circular captures.

4. Discussion and Limitations

DroNeRF demonstrates that a swarm of inexpensive drones can outperform traditional capture methods by being "geometry-aware." By using the object itself for self-localization and pose optimization, the system removes the need for high-end sensors.

However, several limitations remain:

  • Static Scene Constraint: The current algorithm assumes the target is stationary. Movement during the multi-drone capture phase results in significant noise or reconstruction failure.
  • Initial Seed Dependency: The optimization's success is tied to the accuracy of the initial four-image bounding box. Poor initial placement can lead to suboptimal final poses.

5. Conclusion

DroNeRF represents a significant step toward autonomous, high-fidelity 3D scanning in the wild. By replacing uniform sampling with geometry-driven optimization, the framework enables high-quality reconstructions from a remarkably small number of viewpoints. Future work will focus on extending this capability to dynamic scenes and improving the drones' ability to locate subjects in cluttered, unconstrained environments.

Summary updated

BibTeX
@inproceedings{patel2023dronerf,
  author = {Dipam Patel and Phu Pham and Aniket Bera},
  title = {{DroNeRF: Real-time Multi-agent Drone Pose Optimization for Computing Neural Radiance Fields}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2023},
  doi = {10.1109/iros55552.2023.10342420},
}
Figure