AvoidBench: A high-fidelity vision-based obstacle avoidance benchmarking suite for multi-rotors

Hang Yu, Guido C. H. E. de Croon, Christophe De Wagter

Abstract

Obstacle avoidance is a cornerstone of autonomous drone research, yet the field lacks a standardized consensus on testing methodologies. This absence makes it difficult to objectively compare the performance of various avoidance algorithms. We propose AvoidBench, a benchmarking suite designed to evaluate vision-based obstacle avoidance algorithms through a series of rigorous tasks. By leveraging the high-fidelity multi-rotor dynamics of RotorS and the photorealistic virtual environments of Unity3D, AvoidBench facilitates realistic simulated flight experiments. Beyond standard performance tracking, we introduce specific environment metrics to reveal how algorithms handle varying levels of complexity. To demonstrate the utility of the suite, we compare three state-of-the-art algorithms—Ego-planner, MBPlanner, and Agile-autonomy—and validate the observed trends through real-world experiments.

Introduction

In established fields like Computer Vision or Natural Language Processing, researchers rely on gold-standard benchmarks such as ImageNet or GLUE to measure progress. In autonomous drone research, particularly regarding obstacle avoidance, no such standard exists. Testing a drone is inherently more complex than running an algorithm on a static dataset; the drone’s movement actively alters the sensory data it receives.

Furthermore, many existing simulators suffer from a significant "sim-to-real gap." They often lack either the visual fidelity required for vision algorithms to generalize to the real world or the physical realism necessary to account for drone dynamics. AvoidBench addresses these challenges by providing a robust, standardized bridge between simulation and reality.

The AvoidBench Framework

AvoidBench is built upon the Flightmare backbone but introduces several critical components to ensure high-fidelity benchmarking:

  • High-Fidelity Rendering: The suite utilizes Unity3D to create photorealistic scenes. This is essential for vision-based sensors, such as stereo cameras, to produce data that mimics real-world visual noise and lighting conditions.
  • Accurate Physics: By integrating RotorS, AvoidBench ensures that the multi-rotor dynamics are physically grounded, accounting for the constraints and "feel" of a real aircraft.
  • Modular Pipeline: The architecture is divided into three main modules: AvoidLib (a ZeroMQ-based communication bridge), AvoidManage (a state machine that manages individual flight trials), and AvoidMetrics (the core evaluation engine).
  • Procedural Generation: To avoid overfitting to specific maps, the suite uses Poisson Disc Sampling to generate obstacle fields. This ensures a uniform but random distribution of obstacles, allowing researchers to generate thousands of unique test cases automatically.

Quantifying Environmental Complexity

A key contribution of AvoidBench is the shift away from "obstacle density" as a primary measure of difficulty. The authors argue that density is a poor proxy for complexity because it doesn't account for the drone's physical size or the specific arrangement of gaps. Instead, AvoidBench introduces Traversability—a metric that calculates the average free-flight distance a drone can travel within a map. This provides a more accurate representation of the "openness" of a path relative to the agent.

To further bridge the gap between simulation and reality, the suite introduces a Contrast Factor. This mathematical ratio allows researchers to compare the performance of an algorithm in simulation versus the real world, even when flight distances or environments are not identical.

Comparative Analysis of Algorithms

To illustrate the suite's capabilities, three distinct algorithmic philosophies were tested in two primary environments: an Indoor Warehouse (geometric cylinders) and an Outdoor Forest (complex trees and bushes).

  1. Ego-planner (Optimization-based): Demonstrated high safety and success rates. It proved to be highly efficient with low processing times.
  2. MBPlanner (Motion-primitive-based): Also showed high safety levels but was significantly slower in terms of processing due to its redundant map construction process.
  3. Agile-autonomy (Learning-based): While it took more risks and had a lower success rate in dense environments, it achieved the highest Average Goal Velocity, proving its optimization for high-speed flight.

Real-World Validation

The performance trends observed within AvoidBench were validated using a physical drone equipped with an NVIDIA Jetson Xavier NX and a stereo camera. The real-world flight data closely mirrored the simulation results, confirming that AvoidBench’s high-fidelity approach successfully captures the nuances of physical flight and vision-based navigation.

Conclusion and Future Work

AvoidBench provides a necessary foundation for standardized drone benchmarking. While currently focused on static obstacles in forest and warehouse settings, future iterations aim to include dynamic obstacles (such as moving agents) and more diverse environments like urban canyons or caves. By providing a common ground for evaluation, AvoidBench enables the community to more clearly identify the strengths and weaknesses of emerging autonomy algorithms.

Summary updated

Abstract

Obstacle avoidance is an essential topic in the field of autonomous drone research. When choosing an avoidance algorithm, many different options are available, each with their advantages and disadvantages. As there is currently no consensus on testing methods, it is quite challenging to compare the performance between algorithms. In this paper, we propose AvoidBench, a benchmarking suite which can evaluate the performance of vision-based obstacle avoidance algorithms by subjecting them to a series of tasks. Thanks to the high fidelity of multi-rotors dynamics from RotorS and virtual scenes of Unity3D, AvoidBench can realize realistic simulated flight experiments. Compared to current drone simulators, we propose and implement both performance and environment metrics to reveal the suitability of obstacle avoidance algorithms for environments of different complexity. To illustrate AvoidBench's usage, we compare three algorithms: Ego-planner, MBPlanner, and Agile-autonomy. The trends observed are validated with real-world obstacle avoidance experiments.
BibTeX
@inproceedings{yu2023avoidbench,
  author = {Hang Yu and Guido C. H. E. de Croon and Christophe De Wagter},
  title = {{AvoidBench: A high-fidelity vision-based obstacle avoidance benchmarking suite for multi-rotors}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  url = {https://openreview.net/forum?id=wY0Lpxs4h5},
}
Figure