Human-AI Shared Control via Policy Dissection

Quanyi Li, Zhenghao Peng, Haibin Wu, Lan Feng, Bolei Zhou

Overview

Policy Dissection is a novel, non-intrusive method that turns "black-box" reinforcement learning (RL) agents into controllable partners by mapping their internal neural activity to physical behaviors. By identifying and stimulating specific "motor primitives" within a pre-trained network, humans can guide agents through unseen obstacles or trigger complex maneuvers—like backflips or lane changes—without any retraining or model modification.

The Problem with Black-Box RL

Deep Reinforcement Learning has produced agents capable of incredible feats, from autonomous driving to complex robotics. However, these agents are often brittle "black boxes": they struggle to generalize to environments they haven't seen before, and their decision-making processes are notoriously difficult to interpret.

When an RL agent fails in a new environment, developers typically face two expensive choices: retrain the agent with a more complex reward function, or use "Goal-Conditioned RL," which requires designing the agent for controllability from the very beginning. This research addresses a critical gap: how to take an already trained, high-performing policy and give a human the "steering wheel" to help it navigate edge cases or perform new tasks.

Methods: Policy Dissection

Inspired by neuroscience studies where electrical stimulation of a primate's motor cortex elicits specific body movements, the authors developed Policy Dissection. The process follows four key steps:

  1. Monitoring: The researchers roll out a well-trained policy and record both the activations of every hidden unit (neuron) in the neural network and the agent's kinematic attributes (like speed, yaw rate, and height).
  2. Frequency Matching: Instead of looking for simple correlations, the team uses Short-time Fourier Transform (STFT) to analyze the frequency of neural activity versus physical movement. They hypothesize that a unit whose "rhythm" matches a specific kinematic attribute is likely responsible for controlling it.
  3. Building a Stimulation-evoked Map: By identifying these "motor primitives," they create a map detailing which neurons control which behaviors.
  4. Shared Control: During inference, a human can modulate or "stimulate" these units (e.g., via keyboard input) to overwrite the agent's default behavior. This allows a human to intervene only when necessary.
The Policy Dissection pipeline: from monitoring neural activity to human-AI shared control.

Key Insights

  • Emergent Controllability: Perhaps the most surprising finding is that interpretable "motor primitives" emerge naturally during standard RL training, even when the agent isn't explicitly trained to be controllable.
  • Frequency as a Universal Language: Using frequency discrepancy (STFT) allows the researchers to bypass the "phase discrepancy" problem (where a neuron might fire slightly before or after a movement), making it much easier to link abstract neural logic to physical action.
  • Latent Skills: Agents trained for simple tasks often learn "latent" skills. For example, the Cassie robot, trained only to move forward, was found to have internal units that, when stimulated, allowed it to crouch, jump, or even backflip.
  • Causality via Intervention: By manually activating these units and observing the resulting behavior, the authors move beyond mere correlation to prove a causal link between specific neurons and kinematic outcomes.

Key Results

The authors evaluated their method across several diverse simulation environments, including MetaDrive (autonomous driving), IsaacGym (Cassie and ANYmal-C robots), MuJoCo (Ant and Walker), and Pybullet-A1.

  • Superior Generalization: In the MetaDrive driving task, the human-AI shared control system achieved a 95% success rate in unseen, hazardous environments, whereas the AI-only baseline struggled significantly.
  • Zero-Shot Task Transfer: A quadruped robot with only proprioceptive sensors (unable to "see" its surroundings) successfully navigated an obstacle course with human guidance. It effectively "borrowed" the human's vision to trigger its own internal sidestepping primitives.
  • Precision and Robustness: The control precision achieved by stimulating these emergent primitives was comparable to state-of-the-art controllers specifically trained for goal-tracking. Furthermore, the method proved effective across different RL algorithms (PPO, SAC) and varying network depths.
Examples of behaviors like jumping and backflipping triggered via Policy Dissection.

Limitations and Future Work

While promising, the authors note several limitations to the current approach:

  • Input Constraints: The method is currently evaluated on continuous control tasks using state vectors. Extending this to discrete control or vision-based (CNN) policies is a key next step.
  • Theoretical Safety: The study is empirical and lacks theoretical guarantees. In real-world safety-critical systems, a conflict between human input and AI logic could lead to unpredictable risks.
  • Automation: While the "motor primitives" are found automatically, the "stimulation-evoked map" (deciding which primitives to combine for a complex behavior like a backflip) still requires some human trial and error. Future work could involve automating this mapping process.

Summary updated

Abstract

Human-AI shared control allows human to interact and collaborate with autonomous agents to accomplish control tasks in complex environments. Previous Reinforcement Learning (RL) methods attempted goal-conditioned designs to achieve human-controllable policies at the cost of redesigning the reward function and training paradigm. Inspired by the neuroscience approach to investigate the motor cortex in primates, we develop a simple yet effective frequency-based approach called Policy Dissection to align the intermediate representation of the learned neural controller with the kinematic attributes of the agent behavior. Without modifying the neural controller or retraining the model, the proposed approach can convert a given RL-trained policy into a human-controllable policy. We evaluate the proposed approach on many RL tasks such as autonomous driving and locomotion. The experiments show that human-AI shared control system achieved by Policy Dissection in driving task can substantially improve the performance and safety in unseen traffic scenes. With human in the inference loop, the locomotion robots also exhibit versatile controllable motion skills even though they are only trained to move forward. Our results suggest the promising direction of implementing human-AI shared autonomy through interpreting the learned representation of the autonomous agents. Code and demo videos are available at https://metadriverse.github.io/policydissect
BibTeX
@inproceedings{li2022human,
  author = {Quanyi Li and Zhenghao Peng and Haibin Wu and Lan Feng and Bolei Zhou},
  title = {{Human-AI Shared Control via Policy Dissection}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=Z_pTNrD_Xt},
}
Figure