Hovering Flight of Soft-Actuated Insect-Scale Micro Aerial Vehicles using Deep Reinforcement Learning

TL;DR

This paper presents a novel deep reinforcement learning (RL) controller for soft, insect-scale micro aerial vehicles (IMAVs). By combining a modified behavior cloning approach with state-action re-matching and domain randomization, they successfully enable stable hovering flights on real-world robots, overcoming challenges like system delay and model uncertainty. This marks a significant step towards achieving agile flight in these delicate, bio-inspired robots.

What Problem Does This Paper Address?

Insect-scale micro aerial vehicles (IMAVs), especially those with soft actuators like dielectric elastomer actuators (DEAs), offer unique advantages such as resilience to collisions. However, their small size and soft nature introduce significant control challenges. These include:

• Fast Dynamics: IMAVs operate on millisecond timescales, making precise control difficult.
• System Delay: Communication between sensors, control systems, and actuators, along with the inherent response time of soft actuators, introduces delays (around 15-20 ms). This delay is critical for fast-dynamics systems and can lead to instability or poor performance if not accounted for.
• Model Uncertainty: Manufacturing variations in soft actuators and airframes lead to uncertainties in the robot's physical properties (e.g., moment of inertia).
• External Disturbances: Tethers used for power and control can introduce unpredictable forces and torques, further complicating control.

Traditional model-based controllers often resort to conservative gains to handle these issues, which severely limits their performance and ability to execute complex maneuvers. While learning-based methods have shown promise, bridging the "sim-to-real" gap for these complex systems remains a significant hurdle.

Methods

The authors propose a two-stage learning-based controller:

1. Modified Behavior Cloning (BC) for Initialization:
   • Domain-Randomized Expert Demonstrations: To improve robustness against model uncertainty, they generate expert demonstrations using a model-based controller with randomized robot parameters (mass, moment of inertia) and environmental disturbances (actuator delay, external forces/torques).
2. State-Action Re-Matching: To explicitly account for the system delay ($d$), they "re-match" the state-action pairs. This means that an action taken at time $t$ in the simulator is paired with the state observed at time $t+d$ in the real world. This ensures the controller learns to associate the correct action with the state it will actually observe when that action is executed.
3. Supervised Learning: A neural network (NN) controller is trained using these re-matched, domain-randomized demonstrations.
4. Proximal Policy Optimization (PPO) for Fine-Tuning:
   • The BC-initialized policy is then further optimized using PPO in a simulated environment that also incorporates system delay and randomized parameters.
5. A carefully designed reward function is used, penalizing deviations from the desired hovering state (position, velocity, orientation) and also penalizing aggressive, fluctuating control commands to ensure smoother operation and prolong hardware life.

Datasets

The paper does not explicitly mention or introduce new benchmark datasets. Instead, it focuses on generating its own "expert demonstrations" through simulations.

• Generated Expert Demonstrations: These are created from a model-based controller operating in a simulated environment with randomized domain parameters (mass, moment of inertia, external disturbances, and delay). The authors state that this method allows for efficient sampling of diverse state-action pairs without needing iterative dataset aggregation techniques like DAgger.

Key Insights and Ideas

• State-Action Re-Matching is Crucial for Delay Compensation: Explicitly re-matching states and actions to account for system delay is a key innovation that allows the controller to learn effectively in the presence of this critical factor.
• Domain Randomization in BC Enhances Robustness: Introducing parameter variations during the initial behavior cloning phase significantly improves the controller's ability to generalize to real-world uncertainties, rather than relying solely on PPO to fix these issues later.
• Smoothness Matters for Hardware: Penalizing aggressive control commands during PPO fine-tuning is essential for the longevity of soft actuators and the overall robot, even if it slightly impacts theoretical performance metrics in simulation.
• Sim-to-Real for Soft IMAVs is Achievable: The paper demonstrates that a deep RL controller trained in simulation can be successfully deployed on physical soft-actuated IMAVs, achieving stable hovering. This is a significant step in validating the effectiveness of simulation-based training for these complex robots.

Key Results

• Simulation Performance:
  • Modified BC (with state-action re-matching and domain randomization) improved the median reward by 75% compared to baseline BC.
• PPO fine-tuning further enhanced the reward by 62% and reduced thrust fluctuation by 51%.
• Experimental Flight Results:
  • The controller enabled multiple zero-shot stable hovering flights on two distinct soft-actuated IMAVs (720 mg and 850 mg).
• The longest hovering flight achieved was 50 seconds.
• For the 50-second flight, the root-mean-square errors (RMSEs) were 1.34 cm in the lateral direction and 0.05 cm in altitude.
• These performance metrics outperform state-of-the-art long-endurance flights on similar-scale IMAVs.

Limitations and Future Work

• Limitations:
  • The current work focuses on hovering flights. While it's a foundational step, it doesn't yet achieve the agile maneuvers seen in natural insects.
• The robots are tethered for power, limiting their untethered operational time and range.
• The computational capacity of onboard microprocessors for soft IMAVs is constrained, requiring efficient NN architectures.
• Future Work:
  • Extending the controller to perform more complex tasks such as wall perching, inverted ceiling landings, and aggressive trajectory following.
• Developing untethered flight capabilities.
• Exploring more advanced control strategies for achieving insect-like agile maneuverability.