TL;DR
This paper presents the first fully neuromorphic pipeline for controlling a flying drone, enabling it to perform complex maneuvers like hovering, landing, and sideways flight using only event-based vision and a neuromorphic processor. This breakthrough demonstrates the potential of neuromorphic systems for creating smaller, more intelligent, and energy-efficient robots.
What Problem Does This Paper Address?
Traditional deep learning approaches for drone control, while powerful, require significant computational resources, limiting their deployment on small, power-constrained platforms like drones. Neuromorphic hardware, inspired by biological brains, offers a promising alternative due to its asynchronous, sparse, and energy-efficient processing. However, implementing complex tasks like vision-based control on these platforms has been challenging due to difficulties in training spiking neural networks (SNNs) and the limited capabilities of current neuromorphic processors. This paper tackles the problem of creating a complete, end-to-end neuromorphic vision-to-control system for autonomous drone flight, bridging the gap between raw event-based visual input and low-level flight commands.
Methods
The authors developed a two-part neuromorphic pipeline:
- Vision Network (Spiking Neural Network - SNN): This network takes raw event-based camera data (from a DAVIS240C camera) and estimates optical flow for each of the four image corners. It's a five-layer SNN with 28.8k neurons, trained using a self-supervised contrast maximization framework. This approach leverages the temporal dynamics of events to deblur them and estimate motion without explicit ground truth labels during training.
- Control Network (Linear Decoder): This part takes the estimated optical flow from the vision network, along with the drone's absolute roll and pitch, and outputs low-level control commands (thrust, attitude). It was trained using a genetic algorithm in a drone simulator. Crucially, the simulator didn't need to perfectly replicate event-based vision; instead, it used ground-truth pose to generate expected corner flows.
A key innovation is the split training of the vision and control components, allowing for efficient training of each part independently. The entire pipeline was implemented on an Intel Loihi neuromorphic processor, demonstrating real-time performance.
Datasets
- Real event data: The vision network was trained and evaluated on approximately 40 minutes of real event sequences recorded by a downward-facing event-based camera on a drone in an indoor environment. Ground truth pose data was provided at 180 Hz for evaluation.
- Simulated drone data: The control network was trained in a simulated environment (Flightmare) using ground-truth pose to generate visual observables.
Key Insights and Ideas
- End-to-end neuromorphic control is feasible: The paper demonstrates that a fully neuromorphic system can successfully control a flying drone for complex tasks, moving beyond simpler demonstrations.
- Split training is effective: Separating the training of the vision and control modules significantly simplifies the overall training process, especially for the control part which avoids the complexities of simulating event-based vision.
- Self-supervised vision for motion estimation: The use of contrast maximization for training the vision SNN proves robust and effective for estimating optical flow from event data, even without explicit ground truth during training.
- Sim-to-real transfer is successful: The pipeline trained in simulation and deployed on the real robot performed well, highlighting the effectiveness of the training strategies and the robustness of the neuromorphic approach.
- Neuromorphic processing offers significant energy savings: The Loihi processor achieved an execution frequency of 200 Hz while consuming only 27 µJ per inference, a stark contrast to conventional hardware.
Key Results
- The neuromorphic pipeline successfully enabled the drone to perform hovering, landing, and sideways maneuvering, even while yawing simultaneously.
- The system achieved a real-time execution frequency of 200 Hz on the Intel Loihi neuromorphic processor.
- The energy consumption was remarkably low, at 27 µJ per inference.
- The vision network could accurately estimate optical flow, even for fast motion sequences (up to 200 deg/s camera spin).
- The control system demonstrated robustness across various lighting conditions, though performance degraded in extremely dark environments where event data was scarce.
Limitations and Future Work
- Linear controller limitations: The linear nature of the control decoder leads to steady-state errors, particularly when dealing with nonlinear dynamics like drag. The authors suggest that more advanced control strategies or further integration of nonlinear dynamics into the training could improve performance.
- Hardware constraints: The current neuromorphic processor has limitations in terms of neuron and synapse counts, as well as I/O bandwidth, which restricted the complexity of the control network. Future work could explore more advanced neuromorphic hardware with higher capacity.
- Full neuromorphic integration: The current system still relies on a conventional flight controller for low-level actuation. The authors envision a future where the entire drone, including motors, is neuromorphic for maximum efficiency and agility.
- Generalization to more complex environments: The current vision system assumes a static, planar scene. Extending this to dynamic or textured environments would be a significant next step.
Summary updated