Introduction
Unmanned Aerial Vehicles (UAVs) are increasingly reliant on sophisticated Reinforcement Learning (RL) models for autonomous navigation and decision-making. However, these models are computationally intensive, creating a significant conflict with the strict Size, Weight, and Power (SWaP) constraints of small-scale drones. While undervolting—operating hardware at near-threshold voltages—is a standard technique for reducing processor power consumption, it introduces hardware instability in the form of bit errors within SRAM memory.
Traditionally, these errors are viewed as catastrophic failures. BERRY (Bit Error Robustness for Energy-Efficient RL) challenges this paradigm by proposing a framework that embraces hardware unreliability. By training RL agents to be resilient to bit flips, BERRY enables aggressive undervolting that leads to system-wide energy savings, primarily through the reduction of thermal management hardware weight.
The Voltage-Physics Relationship
The primary innovation of BERRY lies in its identification of the "Voltage-Physics Relationship." In autonomous flight, the energy consumed by the propulsion system (flight energy) far outweighs the energy consumed by the onboard computer (processing energy). However, the two are linked by the drone's physical mass.
High-voltage processing generates significant heat, requiring substantial heatsinks to prevent thermal throttling. By lowering the operating voltage, the thermal output of the processor decreases, allowing for a smaller, lighter heatsink. Because flight energy is a function of the drone's total weight, this reduction in payload mass enables higher acceleration and faster mission completion, ultimately reducing the total energy required for a mission. BERRY provides the algorithmic robustness necessary to unlock these physical benefits by ensuring the RL "brain" remains functional even as the underlying hardware becomes prone to errors.
Methodology: Error-Aware Training (EAT)
BERRY utilizes Error-Aware Training (EAT) to prepare RL agents for the stochastic nature of low-voltage hardware. The framework consists of three primary components:
1. Fault Injection Modeling
During the training phase, BERRY simulates bit errors in the neural network’s weights and activations. These are not arbitrary; they are modeled after the error profiles of 14nm FinFET SRAM chips. As voltage drops, the probability of bit flips (0-to-1 and 1-to-0) increases exponentially. BERRY incorporates these profiles to ensure the simulation matches real-world hardware behavior.
2. Robust Gradient Updates
To ensure stability, the framework employs a dual-pass training mechanism. For each iteration, the system performs a "clean" pass (no errors) and a "perturbed" pass (with injected errors). The model parameters are updated using an average of the gradients from both passes. This approach forces the agent to converge on a policy that is not only high-performing under ideal conditions but also resilient to corruption.
3. Two-Tiered Learning Strategy
- Offline Mode: The agent is trained across a wide range of random bit-flip patterns to develop a generalized robustness that works across different chips and voltage levels.
- On-Device Mode: Recognizing that every silicon chip has unique persistent fault patterns, BERRY supports on-device fine-tuning. This allows the drone to adapt its policy to the specific hardware signatures of its own processor while in the field.
Experimental Evaluation
The authors evaluated BERRY using a comprehensive simulation stack involving Unreal Engine and AirSim for realistic UAV dynamics. The study covered 72 autonomous navigation scenarios, such as package delivery, across varying obstacle densities (Sparse, Medium, and Dense). Hardware energy was estimated using SCALE-Sim and Accelergy, focusing on profiles for the Crazyflie (nano UAV) and DJI Tello (micro UAV) platforms.
Key Findings
- Processing and Flight Efficiency: BERRY enabled the processor to operate at $0.77V_{min}$, resulting in a 3.43× reduction in processing energy. More importantly, the reduction in heatsink weight (from 3.26g to 1.22g) contributed to a 15.62% reduction in total flight energy.
- Mission Endurance: The system achieved an 18.51% increase in the number of successful missions possible on a single battery charge.
- Algorithmic Robustness: Under high bit-error rates ($p=1\%$), a standard RL policy's success rate dropped to 33%. In contrast, BERRY maintained a 74.8% success rate, demonstrating its ability to navigate complex environments despite significant hardware noise.
Discussion and Limitations
While BERRY represents a significant step toward hardware-software co-design for autonomous systems, several challenges remain. The energy cost of on-device training must be carefully balanced against the savings gained from undervolting. Additionally, the current framework focuses on SRAM memory errors; future iterations will need to address logic timing errors—faults that occur when signals fail to propagate through logic gates in time due to low voltage. Finally, implementing BERRY in production requires accurate "Voltage-BER" (Bit Error Rate) profiling for each new generation of hardware.
Conclusion
BERRY demonstrates that hardware unreliability does not have to be a barrier to performance. By integrating hardware error models directly into the RL training pipeline, BERRY transforms bit errors from a failure state into a manageable trade-off. This approach unlocks significant system-level energy savings, paving the way for more capable and enduring autonomous drones in SWaP-constrained environments.
Summary updated