Learning to Switch Between Machines and Humans

Vahid Balazadeh Meresht, Abir De, Adish Singla, Manuel Gomez-Rodriguez

Abstract

Reinforcement learning agents have been mostly developed and evaluated under the assumption that they will operate in a fully autonomous manner -- they will take all actions. In this work, our goal is to develop algorithms that, by learning to switch control between agents, allow existing reinforcement learning agents to operate under different automation levels. To this end, we first formally define the problem of learning to switch control among agents in a team via a 2-layer Markov decision process. Then, we develop an online learning algorithm that uses upper confidence bounds on the agents' policies and the environment's transition probabilities to find a sequence of switching policies. The total regret of our algorithm with respect to the optimal switching policy is sublinear in the number of learning steps and, whenever multiple teams of agents operate in a similar environment, our algorithm greatly benefits from maintaining shared confidence bounds for the environments' transition probabilities and it enjoys a better regret bound than problem-agnostic algorithms. Simulation experiments in an obstacle avoidance task illustrate our theoretical findings and demonstrate that, by exploiting the specific structure of the problem, our proposed algorithm is superior to problem-agnostic algorithms.

Introduction

Most reinforcement learning (RL) research assumes "full automation"—the agent takes every single action. However, in high-stakes real-world scenarios like autonomous driving or medical diagnosis, full automation is often precluded by technical, legal, or societal hurdles. This research argues for a middle ground: adjustable autonomy where RL agents and humans share control.

The core challenge lies in determining when to switch control. For instance, if a machine is proficient at highway driving but struggles in heavy city traffic, the system must learn this boundary. Effective algorithms must address unknown human policies, switching costs (cognitive or physical delays), and the need to separate environmental dynamics (the "physics" of the world) from agent policies (the "behavior" of the driver).

Methodology: The 2-Layer MDP Framework

To address these challenges, the authors propose a 2-layer Markov Decision Process (MDP) framework. This structure disentangles the decision of who should be in control from the actions taken by that controller.

The 2-layer MDP framework showing the Switching Layer and Action Layer.

As shown in Figure 1, the framework consists of:

  • Layer 1 (Switching Layer): A "meta-policy" decides which agent (e.g., Human or Machine) should be in control based on the current state and the previous controller (to account for switching costs).
  • Layer 2 (Action Layer): The chosen agent takes an action based on its internal policy, leading to an environment transition.

The proposed algorithm, UCRL2-MC (Upper Confidence Bound for Reinforcement Learning with Multiple Confidence sets), maintains separate confidence intervals for environment transition probabilities and individual agent policies. By applying the "optimism in the face of uncertainty" principle, UCRL2-MC explores switching policies until it converges on an optimal strategy. This disentanglement allows the system to learn environment physics once and apply it across different agent pairings, significantly accelerating learning in multi-team settings.

Experimental Setup

The approach was evaluated using two simulation environments:

  1. RiverSwim: A classic RL benchmark for exploration.
  2. Obstacle Avoidance (Lane Driving): A custom environment where a car navigates three lanes with varying traffic levels.
Obstacle avoidance task with varying traffic densities: (a) no-car, (b) light, and (c) heavy.

In the driving task (Figure 2), the environment includes different cell types (road, grass, stone, car). The team consists of a machine driver (efficient in low traffic) and a human driver (more reliable in heavy traffic but with noisier actions).

Key Results

The theoretical and experimental results highlight several strengths of the UCRL2-MC algorithm:

  • Sublinear Regret: The algorithm's regret grows slower than the number of steps, indicating successful convergence to the optimal switching policy.
  • Multi-Team Efficiency: In settings with multiple teams (e.g., one machine paired with various humans), UCRL2-MC outperformed standard, problem-agnostic algorithms by sharing knowledge of the environment across all teams.
  • Adaptive Control: In the driving task, the algorithm successfully learned to utilize the machine for "no-car" and "light" traffic while proactively switching to the human driver when "heavy" traffic was detected.

Conclusion and Future Work

This work provides a formal foundation for learning to switch control in collaborative AI systems. However, several avenues for future research remain. The current model assumes Markovian human policies, whereas real human behavior is often influenced by non-Markovian factors like fatigue or mood. Additionally, extending the framework from discrete state spaces to continuous environments is necessary for real-world applications. Future deployments must also consider human factors such as reaction times and the need for system transparency during control handovers.

Summary updated

Abstract

Reinforcement learning agents have been mostly developed and evaluated under the assumption that they will operate in a fully autonomous manner -- they will take all actions. In this work, our goal is to develop algorithms that, by learning to switch control between agents, allow existing reinforcement learning agents to operate under different automation levels. To this end, we first formally define the problem of learning to switch control among agents in a team via a 2-layer Markov decision process. Then, we develop an online learning algorithm that uses upper confidence bounds on the agents' policies and the environment's transition probabilities to find a sequence of switching policies. The total regret of our algorithm with respect to the optimal switching policy is sublinear in the number of learning steps and, whenever multiple teams of agents operate in a similar environment, our algorithm greatly benefits from maintaining shared confidence bounds for the environments' transition probabilities and it enjoys a better regret bound than problem-agnostic algorithms. Simulation experiments in an obstacle avoidance task illustrate our theoretical findings and demonstrate that, by exploiting the specific structure of the problem, our proposed algorithm is superior to problem-agnostic algorithms.
BibTeX
@misc{meresht2020learning,
  author = {Vahid Balazadeh Meresht and Abir De and Adish Singla and Manuel Gomez-Rodriguez},
  title = {{Learning to Switch Between Machines and Humans}},
  year = {2020},
  eprint = {2002.04258},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2002.04258},
}
Figure