User-Conditioned Neural Control Policies for Mobile Robotics

Davide Scaramuzza

TL;DR

This paper introduces a novel framework for conditioning neural network-based robot controllers with auxiliary user inputs, allowing for dynamic adjustments during operation. By integrating a Feature-wise Linear Modulation (FiLM) layer, they enable a single quadrotor policy to achieve near time-optimal performance across its entire operational envelope, adapting its agility and viewing direction based on user commands. This work bridges the gap between the robustness of learned controllers and the flexibility of traditional methods, paving the way for safer deployment of AI in real-world robotics.

What Problem Does This Paper Address?

Modern mobile robots, especially agile platforms like quadrotors, often rely on learned neural network controllers that excel at pushing performance limits. However, these controllers are typically rigid, trained for a specific task and lacking the adaptability of classical methods. For instance, while a model-predictive controller (MPC) can easily adjust parameters like maximum acceleration on the fly, a standard neural controller cannot. This inflexibility limits their real-world applicability, as dynamic adjustments are often necessary to handle varying conditions or user intentions. The paper addresses this by enabling users to dynamically influence learned control policies, making them more versatile and robust for deployment outside controlled lab environments.

Methods

The core contribution is a framework that conditions neural control policies on auxiliary inputs using a Feature-wise Linear Modulation (FiLM) layer. This layer, inserted between existing layers of a neural network, applies an affine transformation to intermediate activations, modulated by the conditioning signal. The authors trained quadrotor control policies using model-free reinforcement learning (specifically Proximal Policy Optimization - PPO) for the task of time-optimal navigation through waypoints.

They explored several conditioning architectures:

  • Naive concatenation: Appending the conditioning signal to the observation.
  • Multi-head networks: Using separate network heads for discrete conditioning signals.
  • FiLM-based approaches: Including their proposed augmented FiLM architecture (FiLM*) which also feeds the conditioning signal directly into the control policy.

The conditioning signals investigated were the maximum thrust-to-weight ratio (TWR), controlling the robot's agility, and the viewing direction relative to the next waypoint, influencing perception.

Datasets

The paper does not explicitly mention using pre-existing datasets for training. Instead, the control policies were trained in a custom-built quadrotor simulator. The evaluation was performed both in this simulator and on a real-world agile quadrotor platform with specifications listed in Table I. The experiments involved racing on three different track layouts: a square track, a figure-8 track, and a complex 3D track (Split-S).

Key Insights and Ideas

  • FiLM is a powerful tool for conditioning: The FiLM architecture, particularly the augmented version (FiLM*), significantly outperforms naive concatenation and multi-head approaches in terms of both performance and robustness when conditioning neural controllers.
  • Bridging the gap between learned and classical control: The FiLM*-c policy achieved performance very close to (less than 0.6% slower in simulation, 2% in real-world) specifically trained "fixed-TWR" policies, demonstrating that dynamic conditioning doesn't necessitate a significant performance trade-off.
  • Continuous conditioning is superior: For both naive and FiLM architectures, using continuous conditioning signals (like TWR) yielded better results than discrete, one-hot encoded signals.
  • Agility and perception are controllable: Users can effectively regulate the quadrotor's agility (via TWR) and its viewing direction, enabling adaptive flight strategies.
  • Generalization across tasks and platforms: The FiLM*-c approach showed consistent performance across different track complexities and successfully transferred from simulation to real-world deployment.

Key Results

  • The FiLM*-c architecture achieved the best performance, with lap times less than 0.6% slower than fixed-TWR policies in simulation and 2% slower in real-world experiments.
  • The FiLM*-c policy could operate across the entire TWR range (1.6 to 4.5), whereas other architectures, including naive ones, crashed at lower TWR settings.
  • For the viewing direction conditioning task, the FiLM*-c policy maintained low heading errors across desired viewing directions, outperforming policies trained for a single direction.
  • The network size ablation study indicated that a moderate network size (128/512 neurons for policy/value networks) offered the best balance between expressiveness and trainability.
  • The paper demonstrated that the quadrotor could reach speeds up to 60 km/h and accelerations of 4.5 g.

Limitations and Future Work

  • Simulation-to-Real Gap: While the paper demonstrates successful transfer, the authors acknowledge that the quadratic propeller model used in simulation does not account for all aerodynamic effects, potentially widening the sim-to-real gap. Future work could involve more sophisticated aerodynamic models.
  • Limited Conditioning Signals: The current work focuses on TWR and viewing direction. Future research could explore conditioning on a wider range of user intents or environmental factors.
  • Task Complexity: While tested on racing, the applicability to other complex tasks like manipulation or exploration could be further investigated.
  • Human Intent Interpretation: The current conditioning is explicit (e.g., setting a TWR value). Future work could explore inferring human intent from more implicit signals like natural language or gestures.

Summary updated

Abstract

Recently, learning-based controllers have been shown to push mobile robotic systems to their limits and provide the robustness needed for many real-world applications. However, only classical optimization-based control frameworks offer the inherent flexibility to be dynamically adjusted during execution by, for example, setting target speeds or actuator limits.We present a framework to overcome this shortcoming of neural controllers by conditioning them on an auxiliary input. This advance is enabled by including a feature-wise linear modulation layer (FiLM). We use model-free reinforcementlearning to train quadrotor control policies for the task of navigating through a sequence of waypoints in minimum time. By conditioning the policy on the maximum available thrust or the viewing direction relative to the next waypoint, a user can regulate the aggressiveness of the quadrotor’s flight during deployment. We demonstrate in simulation and in real-world experiments that a single control policy can achieve close to time-optimal flight performance across the entire performance envelope of the robot, reaching up to 60 km/h and 4.5 g in acceleration. The ability to guide a learned controller during task execution has implications beyond agile quadrotor flight, as conditioning the control policy on human intent helps safely bringing learning based systems out of the well-defined laboratory environment into the wild.
BibTeX
@inproceedings{scaramuzza2023user,
  author = {Davide Scaramuzza},
  title = {{User-Conditioned Neural Control Policies for Mobile Robotics}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  doi = {10.1109/icra48891.2023.10160851},
}
Figure