Fully neuromorphic vision and control for autonomous drone flight

Federico Paredes-Vallés, Jesse J. Hagenaars, Julien Dupeyroux, Stein Stroobants, Yingfu Xu, Guido de Croon

TL;DR

Researchers have developed the first fully neuromorphic vision-to-control pipeline for autonomous drone flight, running entirely on Intel’s Loihi neuromorphic processor. By combining a spiking neural network (SNN) for ego-motion estimation with an evolved control layer, the system achieves high-frequency (200 Hz) autonomous flight with extreme energy efficiency—consuming just 27 µJ per inference.

What Problem Does This Paper Address?

Autonomous flight for small drones is a massive challenge in "SWaP" (Size, Weight, and Power). Traditional deep learning relies on power-hungry GPUs and large artificial neural networks (ANNs) that are often too heavy for tiny robots.

Neuromorphic engineering offers a biological alternative: event-based cameras that only transmit changes in brightness (sparse, asynchronous data) and spiking neural networks (SNNs) that mimic the energy-efficient processing of biological brains. However, training SNNs is notoriously difficult because their spiking nature is non-differentiable, making standard backpropagation impossible. Furthermore, current neuromorphic hardware has strict limits on network size, previously restricting these systems to very basic, low-dimensional tasks.

Methods

The authors propose a split pipeline that decouples vision from control, allowing each to be trained using the most effective method for its domain:

  1. Vision SNN: A five-layer SNN (28.8k neurons) maps raw event data to ego-motion (optical flow) estimates. To stay within Loihi's resource limits, the network only processes 16x16 pixel crops from the four corners of the image. It is trained via self-supervised learning using a "contrast maximization" framework, which essentially learns to "deblur" event streams to find the most accurate motion vectors.
  2. Control Layer: A single linear decoding layer converts the vision spikes into low-level control commands (thrust, pitch, roll, and yaw). This was trained in the Flightmare drone simulator using a genetic algorithm. By evolving the controller in simulation, the authors avoided the risks of real-world crashes while bypassing the need to accurately simulate complex event-camera data.
  3. Hardware Integration: The system uses a DAVIS240C event camera and an Intel Loihi (Kapoho Bay) processor. Because the control layer is linear, the authors could "merge" the vision decoding and control encoding into a single matrix multiplication, keeping the entire pipeline on-chip and minimizing latency.
Fig. 1. Overview of the proposed system. (A) Quadrotor used in this work (total weight 1.0 kg, tip-to-tip diameter 35 cm). (B) Hardware overview showing the communication between event-camera, neuromorphic processor, single-board computer and flight controller (C) Pipeline overview showing events as input, processing by the vision network and decoding into a control command. (D) Demonstration of the system for an optical flow divergence landing.

Datasets

  • Vision Training: The authors recorded 40 minutes of real-world event data using a drone flying over a static, planar indoor environment. This was split into 25 minutes for training and 15 minutes for evaluation.
  • Ground Truth: High-precision pose data (180 Hz) was captured via an OptiTrack motion capture system to evaluate the accuracy of the SNN’s ego-motion estimates.
  • Control Training: Conducted entirely within the Flightmare simulator using a modified quadrotor dynamics model.

Key Insights and Ideas

  • Corner-Based Processing: Instead of processing the full high-dimensional frame, the authors realized that ego-motion for a downward-facing camera over a flat surface can be accurately estimated using just the image corners. This drastically reduced the required neuron count.
  • Sim-to-Real via Abstraction: By training the vision SNN on real data and the controller on simulated "perfect" motion vectors, they successfully bridged the "reality gap." The controller learns the physics of flight, while the vision network learns the noise and nuances of real sensors.
  • Linear Merging: The insight that the final decoding of the SNN and the control mapping are both linear allowed them to be mathematically combined. This ensures that the "vision-to-control" step happens entirely within the neuromorphic hardware without needing to move data back to a traditional CPU.
Overview of the spiking vision network architecture and the self-supervised training process.

Key Results

  • Energy Efficiency: The Loihi processor consumed only 27 µJ per inference. For comparison, an NVIDIA Jetson Nano running a similar SNN in software consumed roughly 75,000 µJ per inference—a difference of three orders of magnitude.
  • High Bandwidth: The pipeline achieved an execution frequency of 200 Hz, which is critical for the stability of agile flying robots.
  • Flight Performance: The drone successfully performed hovering, vertical landing (following an optical flow divergence setpoint), and complex lateral maneuvers. It even maintained stability during "frisbee-like" flights where it spun around its yaw axis while moving in a straight line.

Limitations and Future Work

The current system relies on a static, planar surface assumption. If the ground is not flat or if there are moving objects in the scene, the ego-motion estimation would likely fail.

Additionally, the authors noted that the linear controller has limitations, such as steady-state errors at high speeds due to non-linear aerodynamic drag. Future work could involve training more complex, non-linear SNN controllers. Finally, the authors pointed out that current neuromorphic hardware is often bottlenecked by I/O (e.g., USB interfaces); moving toward direct sensor-to-processor interfaces (like AER) would further unlock the low-latency potential of these systems.

Summary updated

Abstract

Biological sensing and processing is asynchronous and sparse, leading to low-latency and energy-efficient perception and action. In robotics, neuromorphic hardware for event-based vision and spiking neural networks promises to exhibit similar characteristics. However, robotic implementations have been limited to basic tasks with low-dimensional sensory inputs and motor actions due to the restricted network size in current embedded neuromorphic processors and the difficulties of training spiking neural networks. Here, we present the first fully neuromorphic vision-to-control pipeline for controlling a freely flying drone. Specifically, we train a spiking neural network that accepts high-dimensional raw event-based camera data and outputs low-level control actions for performing autonomous vision-based flight. The vision part of the network, consisting of five layers and 28.8k neurons, maps incoming raw events to ego-motion estimates and is trained with self-supervised learning on real event data. The control part consists of a single decoding layer and is learned with an evolutionary algorithm in a drone simulator. Robotic experiments show a successful sim-to-real transfer of the fully learned neuromorphic pipeline. The drone can accurately follow different ego-motion setpoints, allowing for hovering, landing, and maneuvering sideways$\unicode{x2014}$even while yawing at the same time. The neuromorphic pipeline runs on board on Intel's Loihi neuromorphic processor with an execution frequency of 200 Hz, spending only 27 $\unicode{x00b5}$J per inference. These results illustrate the potential of neuromorphic sensing and processing for enabling smaller, more intelligent robots.
BibTeX
@article{paredesvalles2024fully,
  author = {Federico Paredes-Vallés and Jesse J. Hagenaars and Julien Dupeyroux and Stein Stroobants and Yingfu Xu and Guido de Croon},
  title = {{Fully neuromorphic vision and control for autonomous drone flight}},
  journal = {Sci. Robotics},
  year = {2024},
  doi = {10.1126/scirobotics.adi0591},
}
Figure