Finding 3D Positions of Distant Objects from Noisy Camera Movement and Semantic Segmentation Sequences

Julius Pesonen, Arno Solin, Eija Honkavaara

Introduction

Locating distant objects in 3D space is a critical challenge for aerial robotics, particularly in safety-critical applications like wildfire monitoring. Traditional depth-sensing technologies often fall short at long ranges: LiDAR resolution degrades cubically with distance, and stereo cameras require impractically large baselines to achieve meaningful depth perception at distances exceeding several kilometers.

While Structure-from-Motion (SfM) offers a potential solution using a single moving camera, its high computational demands make it unsuitable for the resource-constrained "edge" hardware typically found on small drones. Furthermore, SfM relies on distinct visual keypoints, which are absent in amorphous, shifting targets like smoke plumes. This research proposes a lightweight alternative for long-range 3D localization using 2D semantic segmentation and probabilistic filtering.

Conceptual overview of the 3D localization system using a moving drone camera to track distant smoke plumes.

Methodology

The proposed system translates 2D image detections into 3D coordinates by integrating two mathematical frameworks:

  1. Multi-view Triangulation (MVT): This geometric approach utilizes multiple viewpoints captured as the drone traverses its path. Rays are projected from the camera center through the centroid of the detected object mask. The 3D position is estimated at the point of near-intersection. To handle noise from GNSS (GPS) data or false detections, the authors employ Robust Multi-view Triangulation (RMVT), which incorporates RANSAC (Random Sample Consensus) to filter out outlier frames.
  2. Particle Filters (PF): This probabilistic method maintains a "cloud" of thousands of potential object locations (particles). As new visual evidence is acquired, the system updates the weights of these particles. Particles inconsistent with the visual data are discarded, while new ones are generated near high-probability locations. This approach is particularly effective at representing the inherent uncertainty and the physical shape of the target.

Experimental Evaluation

The methods were validated across three scenarios:

  • Synthetic Simulation: A controlled environment used to test resilience against pose errors and detection noise.
  • Telecom Mast Sequence: Real-world tracking of a stationary tower to establish a baseline for point-target accuracy.
  • Industrial Smoke Sequence: Tracking smoke from a chimney at a distance of 1,770 meters. This sequence utilized the Segment Anything Model (SAM 3) for initial 2D segmentation, testing the system's ability to handle amorphous shapes.

Key Results and Discussion

The study highlights a trade-off between precision and stability. In synthetic tests with high noise, RMVT achieved a minimum Root Mean Square Error (RMSE) of approximately 1.2 meters, significantly lower than the Particle Filter's ~17 meters. However, the Particle Filter demonstrated superior stability over time and better resilience to "dirty" data, such as false positive detections.

Comparison of RMSE convergence for Particle Filter, Multi-view Triangulation, and Robust Multi-view Triangulation over drone translation distance.

A critical finding is the "convergence requirement." As shown in the performance metrics, the drone must travel a baseline of approximately 150–200 meters before the 3D estimates for a target 2km away become reliable. In real-world smoke tracking, the system achieved an RMSE of 200–350 meters. While less precise than point-target tracking, this level of accuracy is sufficient for directing emergency services to specific forest quadrants.

Conclusion and Future Work

The developed methods are computationally efficient enough to run on embedded hardware like the NVIDIA Jetson Orin NX, enabling fully autonomous onboard localization. Current limitations include the assumption of a single target; future work will focus on multi-target tracking to handle multiple simultaneous smoke plumes. Additionally, incorporating digital elevation models to constrain the initial search area of the Particle Filter could significantly accelerate convergence.

Summary updated

Abstract

3D object localisation based on a sequence of camera measurements is essential for safety-critical surveillance tasks, such as drone-based wildfire monitoring. Localisation of objects detected with a camera can typically be solved with specialised sensor configurations or 3D scene reconstruction. However, in the context of distant objects or tasks limited by the amount of available computational resources, neither solution is feasible. In this paper, we show that the task can be solved with either multi-view triangulation or particle filters, with the latter also providing shape and uncertainty estimates. We studied the solutions using 3D simulation and drone-based image segmentation sequences with global navigation satellite system (GNSS) based camera pose estimates. The results suggest that combining the proposed methods with pre-existing image segmentation models and drone-carried computational resources yields a reliable system for drone-based wildfire monitoring. The proposed solutions are independent of the detection method, also enabling quick adaptation to similar tasks.
BibTeX
@misc{pesonen2025finding,
  author = {Julius Pesonen and Arno Solin and Eija Honkavaara},
  title = {{Finding 3D Positions of Distant Objects from Noisy Camera Movement and Semantic Segmentation Sequences}},
  year = {2025},
  eprint = {2509.20906},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.20906},
}
Figure