TL;DR
AvoidBench is a new high-fidelity benchmarking suite designed to standardize how we evaluate vision-based obstacle avoidance for drones. By pairing photo-realistic Unity3D environments with accurate RotorS flight dynamics, it provides a robust "sim-to-real" bridge that allows researchers to objectively compare different autonomy algorithms across varying levels of environmental complexity.
What Problem Does This Paper Address?
In fields like Computer Vision or NLP, researchers have gold-standard benchmarks like ImageNet or GLUE to measure progress. In autonomous drone research, specifically obstacle avoidance, such a consensus doesn't exist.
Testing a drone isn't as simple as running an algorithm on a static dataset; the drone’s own movements change the sensory data it receives. Furthermore, many existing simulators suffer from a "sim-to-real gap"—they either have poor visual fidelity (making vision algorithms fail in the real world) or unrealistic physics. This makes it nearly impossible to say with certainty whether Algorithm A is truly "better" than Algorithm B, or if it just performed well because the test environment was too simple.
Methods
AvoidBench is built on the Flightmare backbone but introduces several critical components to ensure high-fidelity benchmarking:
- The Rendering Engine: It uses Unity3D to create photo-realistic scenes, which is crucial for vision-based sensors (like stereo cameras) to behave as they would in the real world.
- The Physics Engine: It integrates RotorS for accurate multi-rotor dynamics, ensuring the drone's "feel" and constraints are realistic.
- The Pipeline: The suite is modular, consisting of AvoidLib (a ZeroMQ-based communication bridge), AvoidManage (the state machine managing the flight trials), and AvoidMetrics (the evaluation engine).
- Procedural Generation: Instead of static maps, it uses Poisson Disc Sampling to auto-generate obstacle fields. This ensures a uniform but random distribution of obstacles, allowing for thousands of unique test cases.
Datasets
AvoidBench doesn't use a static image dataset; instead, it provides two primary procedural environments:
- Indoor Warehouse: Features geometrical obstacles (cylinders) with varying textures and opacities.
- Outdoor Forest: A complex environment containing "red trees" and non-geometrical "bushes" on undulating terrain.
The authors evaluated three state-of-the-art algorithms within these environments: Ego-planner (optimization-based), MBPlanner (motion-primitive-based), and Agile-autonomy (learning-based).
Key Insights and Ideas
- Traversability as a Metric: The authors argue that "obstacle density" is a poor way to measure difficulty. Instead, they use Traversability—a metric that calculates the average free-flight distance a drone can travel in a map. This accounts for the drone's physical size and the actual "openness" of the path.
- The Contrast Factor: To solve the sim-to-real comparison problem, they introduced a "Contrast Factor." This mathematical ratio allows researchers to compare the performance of the same algorithm in simulation versus the real world, even if the flight distances aren't identical.
- Safety vs. Speed Trade-offs: The benchmark reveals that different algorithmic "philosophies" yield different strengths. Optimization-based methods might be safer, while learning-based methods might prioritize speed.
Key Results
- Safety Performance: In the forest and warehouse tests, Ego-planner and MBPlanner showed higher success rates (safety) compared to Agile-autonomy.
- Agility: True to its name, Agile-autonomy achieved the highest Average Goal Velocity, proving it is optimized for high-speed flight even if it takes more risks.
- Efficiency: Ego-planner and Agile-autonomy had the lowest processing times. MBPlanner was significantly slower due to its redundant map construction process.
- Real-World Validation: The authors built a physical drone (using an NVIDIA Jetson Xavier NX and a stereo camera) and flew it in real-world environments. The performance trends observed in AvoidBench closely matched the real-world results, validating the simulator's high fidelity.
Limitations and Future Work
While AvoidBench is a significant step forward, the authors note a few areas for growth:
- Environment Diversity: Currently, the suite only features two main scenes (forest and warehouse). Expanding to urban environments or caves would increase its utility.
- Dynamic Obstacles: The current benchmark focuses on static obstacles. Future iterations could include moving objects (like birds or other drones) to test reactive avoidance.
- Metric Linearity: The authors noted that in very complex outdoor scenes with bushes, the relationship between "Relative Gap Size" and "Traversability" isn't always linear, suggesting that even more sophisticated environment metrics may be needed.
Summary updated