FAST-Dynamic-Vision: Detection and Tracking Dynamic Objects with Event and Depth Sensing

Botao He, Haojia Li, Siyuan Wu, Dong Wang, Zhiwei Zhang, Qianli Dong, Chao Xu, Fei Gao

TL;DR

FAST-Dynamic-Vision is a perception system that allows autonomous drones to detect and track high-speed objects (moving at over 10 m/s) with millisecond-level latency. By fusing the high temporal resolution of event cameras with the spatial data of depth sensors, the system enables UAVs to accurately predict the 3D trajectories of incoming obstacles, providing the foundation for agile dodging maneuvers.

What Problem Does This Paper Address?

For an autonomous drone, dodging a rock or a ball thrown at high speed is a massive technical hurdle. Traditional "frame-based" cameras are often too slow; they require tens of milliseconds for exposure and suffer from motion blur. By the time a standard vision pipeline processes a sequence of frames to predict a trajectory, a fast-moving object may have already traveled several meters, leaving the drone no time to react.

While event cameras (which only report per-pixel intensity changes asynchronously) offer microsecond-level resolution and no motion blur, they have two major drawbacks:

  1. Ego-motion noise: When the drone moves, the entire scene generates "events," making it hard to distinguish a moving object from the static background.
  2. Scale ambiguity: A single event camera cannot easily determine how far away an object is or how large it is.

Methods

The authors propose a three-stage pipeline to solve these issues:

  1. Advanced Ego-Motion Compensation: Unlike previous methods that only account for camera rotation, this system uses IMU data and depth information to compensate for both rotational and translational motion. This effectively "cancels out" events caused by the drone’s own flight, leaving only the events generated by the moving object.
  2. Iterative Gaussian Fitting for Detection: The system represents events using a "mean-time image" (where pixel values represent average timestamps). They use an adaptive thresholding method—varying based on the drone's velocity—and an iterative Gaussian fitting algorithm to quickly identify the Region of Interest (ROI) containing the dynamic object.
  3. Asynchronous Event-Depth Fusion: To predict a 3D trajectory, the authors use a non-linear optimization framework. Because event cameras are much faster than depth cameras, the sensors aren't synchronized. The system treats them as independent observations of a parabolic (free-fall) physical model, fusing high-frequency 2D positions from the event camera with lower-frequency depth samples to recover the object's full 3D path.

Datasets

The researchers validated their system through extensive real-world experiments:

  • Hardware: A customized quadrotor equipped with an iniVation DVXplorer event camera, an Intel RealSense D435i depth camera, and a DJI Manifold 2-C onboard computer.
  • Scenarios: Indoor and outdoor "ball-throwing" tests with varying ball sizes, lighting conditions (from 8 to 1100 lux), and drone speeds (over 5 m/s).
  • Ground Truth: A Vicon motion capture system was used to provide high-accuracy reference data for the ball's trajectory.
  • Benchmarks: The system was compared against existing event-based algorithms like BetterFlow and Falanga et al.

Key Insights and Ideas

  • Translation Matters: Most event-based systems ignore camera translation because it's computationally expensive to solve without depth. By leveraging an onboard depth sensor, this paper shows that translational compensation is vital for detecting objects while the drone is flying fast.
  • The "Mean-Time Image" Advantage: Using the average timestamp of events in a pixel (rather than just counting events) makes it much easier to segment objects based on their relative speed on the image plane.
  • Physical Priors as a Bridge: By assuming the object follows a standard gravity-based trajectory (Equation 11 in the paper), the authors can bridge the gap between two sensors running at completely different frequencies without needing hardware-level synchronization.

Key Results

  • Superior Contrast: In high-speed scenarios, the proposed ego-motion compensation achieved a "Relative Contrast" of 51.4%, compared to just 3.9% for previous state-of-the-art methods. This makes the object much easier for the detection algorithm to "see."
  • Accuracy Boost: In "Fast Forward" flight tests, the fusion-based trajectory estimation reduced the Root Mean Square Error (RMSE) to 0.201m, a massive improvement over the 0.784m error seen in monocular event-only methods.
  • Real-Time Performance: The entire pipeline runs onboard the drone with low enough latency to support dodging objects moving at 10+ meters per second.

Limitations and Future Work

The primary limitation is the physical model assumption: the current trajectory estimator assumes the object is in free-fall (parabolic motion). This works for rocks or balls but would likely fail for "smart" dynamic objects like other drones or birds that can change direction mid-flight.

The authors also note that while their method is more robust, it is slightly more computationally demanding than the simplest event-processing methods. Future work will focus on integrating this perception system with active motion planning to close the loop on fully autonomous high-speed obstacle avoidance.

Summary updated

Abstract

The development of aerial autonomy has enabled aerial robots to fly agilely in complex environments. However, dodging fast-moving objects in flight remains a challenge, limiting the further application of unmanned aerial vehicles (UAVs). The bottleneck of solving this problem is the accurate perception of rapid dynamic objects. Recently, event cameras have shown great potential in solving this problem. This paper presents a complete perception system including ego-motion compensation, object detection, and trajectory prediction for fast-moving dynamic objects with low latency and high precision. Firstly, we propose an accurate ego-motion compensation algorithm by considering both rotational and translational motion for more robust object detection. Then, for dynamic object detection, an event camera-based efficient regression algorithm is designed. Finally, we propose an optimization-based approach that asynchronously fuses event and depth cameras for trajectory prediction. Extensive real-world experiments and benchmarks are performed to validate our framework. Moreover, our code will be released to benefit related researches.
BibTeX
@inproceedings{he2021fast,
  author = {Botao He and Haojia Li and Siyuan Wu and Dong Wang and Zhiwei Zhang and Qianli Dong and Chao Xu and Fei Gao},
  title = {{FAST-Dynamic-Vision: Detection and Tracking Dynamic Objects with Event and Depth Sensing}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=JnZkCvVfQ2},
}

Loading related content...

Figure