AvoidBench: A high-fidelity vision-based obstacle avoidance benchmarking suite for multi-rotors

Hang Yu, Guido C. H. E. de Croon, Christophe De Wagter

TL;DR

AvoidBench is a new high-fidelity benchmarking suite designed to standardize how we evaluate vision-based obstacle avoidance for drones. By pairing photo-realistic Unity3D environments with accurate RotorS flight dynamics, it provides a robust "sim-to-real" bridge that allows researchers to objectively compare different autonomy algorithms across varying levels of environmental complexity.

What Problem Does This Paper Address?

In fields like Computer Vision or NLP, researchers have gold-standard benchmarks like ImageNet or GLUE to measure progress. In autonomous drone research, specifically obstacle avoidance, such a consensus doesn't exist.

Testing a drone isn't as simple as running an algorithm on a static dataset; the drone’s own movements change the sensory data it receives. Furthermore, many existing simulators suffer from a "sim-to-real gap"—they either have poor visual fidelity (making vision algorithms fail in the real world) or unrealistic physics. This makes it nearly impossible to say with certainty whether Algorithm A is truly "better" than Algorithm B, or if it just performed well because the test environment was too simple.

Methods

AvoidBench is built on the Flightmare backbone but introduces several critical components to ensure high-fidelity benchmarking:

  • The Rendering Engine: It uses Unity3D to create photo-realistic scenes, which is crucial for vision-based sensors (like stereo cameras) to behave as they would in the real world.
  • The Physics Engine: It integrates RotorS for accurate multi-rotor dynamics, ensuring the drone's "feel" and constraints are realistic.
  • The Pipeline: The suite is modular, consisting of AvoidLib (a ZeroMQ-based communication bridge), AvoidManage (the state machine managing the flight trials), and AvoidMetrics (the evaluation engine).
  • Procedural Generation: Instead of static maps, it uses Poisson Disc Sampling to auto-generate obstacle fields. This ensures a uniform but random distribution of obstacles, allowing for thousands of unique test cases.

Datasets

AvoidBench doesn't use a static image dataset; instead, it provides two primary procedural environments:

  1. Indoor Warehouse: Features geometrical obstacles (cylinders) with varying textures and opacities.
  2. Outdoor Forest: A complex environment containing "red trees" and non-geometrical "bushes" on undulating terrain.

The authors evaluated three state-of-the-art algorithms within these environments: Ego-planner (optimization-based), MBPlanner (motion-primitive-based), and Agile-autonomy (learning-based).

Key Insights and Ideas

  • Traversability as a Metric: The authors argue that "obstacle density" is a poor way to measure difficulty. Instead, they use Traversability—a metric that calculates the average free-flight distance a drone can travel in a map. This accounts for the drone's physical size and the actual "openness" of the path.
  • The Contrast Factor: To solve the sim-to-real comparison problem, they introduced a "Contrast Factor." This mathematical ratio allows researchers to compare the performance of the same algorithm in simulation versus the real world, even if the flight distances aren't identical.
  • Safety vs. Speed Trade-offs: The benchmark reveals that different algorithmic "philosophies" yield different strengths. Optimization-based methods might be safer, while learning-based methods might prioritize speed.

Key Results

  • Safety Performance: In the forest and warehouse tests, Ego-planner and MBPlanner showed higher success rates (safety) compared to Agile-autonomy.
  • Agility: True to its name, Agile-autonomy achieved the highest Average Goal Velocity, proving it is optimized for high-speed flight even if it takes more risks.
  • Efficiency: Ego-planner and Agile-autonomy had the lowest processing times. MBPlanner was significantly slower due to its redundant map construction process.
  • Real-World Validation: The authors built a physical drone (using an NVIDIA Jetson Xavier NX and a stereo camera) and flew it in real-world environments. The performance trends observed in AvoidBench closely matched the real-world results, validating the simulator's high fidelity.

Limitations and Future Work

While AvoidBench is a significant step forward, the authors note a few areas for growth:

  • Environment Diversity: Currently, the suite only features two main scenes (forest and warehouse). Expanding to urban environments or caves would increase its utility.
  • Dynamic Obstacles: The current benchmark focuses on static obstacles. Future iterations could include moving objects (like birds or other drones) to test reactive avoidance.
  • Metric Linearity: The authors noted that in very complex outdoor scenes with bushes, the relationship between "Relative Gap Size" and "Traversability" isn't always linear, suggesting that even more sophisticated environment metrics may be needed.

Summary updated

Abstract

Obstacle avoidance is an essential topic in the field of autonomous drone research. When choosing an avoidance algorithm, many different options are available, each with their advantages and disadvantages. As there is currently no consensus on testing methods, it is quite challenging to compare the performance between algorithms. In this paper, we propose AvoidBench, a benchmarking suite which can evaluate the performance of vision-based obstacle avoidance algorithms by subjecting them to a series of tasks. Thanks to the high fidelity of multi-rotors dynamics from RotorS and virtual scenes of Unity3D, AvoidBench can realize realistic simulated flight experiments. Compared to current drone simulators, we propose and implement both performance and environment metrics to reveal the suitability of obstacle avoidance algorithms for environments of different complexity. To illustrate AvoidBench's usage, we compare three algorithms: Ego-planner, MBPlanner, and Agile-autonomy. The trends observed are validated with real-world obstacle avoidance experiments. Code is available at: https://github.com/tudelft/AvoidBench
BibTeX
@inproceedings{yu2023avoidbench,
  author = {Hang Yu and Guido C. H. E. de Croon and Christophe De Wagter},
  title = {{AvoidBench: A high-fidelity vision-based obstacle avoidance benchmarking suite for multi-rotors}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  url = {https://openreview.net/forum?id=wY0Lpxs4h5},
}

Loading related content...

Figure