Collaborative Decision Making Using Action Suggestions

Dylan Asmar, Mykel Kochenderfer

Overview

This paper introduces a novel framework for human-AI collaboration where an autonomous agent treats human "action suggestions" as independent observations to update its internal belief state. By using its own policy to "back-calculate" why a human might suggest a specific move, the agent achieves higher performance with fewer human interventions and remains robust even when the human provides suboptimal advice.

The Challenge of Human-AI Handovers

Autonomous systems are becoming more capable, but they remain "brittle"—they often fail in edge cases or unstructured environments where humans excel. The traditional solution is a "human-on-the-loop" structure, where a human monitors the system and takes direct control when things go wrong.

However, direct handover can be jarring, risky, and inefficient. The authors identify a gap: how can an agent benefit from human expertise without the human having to take the wheel? Specifically, they look at Partially Observable Markov Decision Processes (POMDPs), where the agent is uncertain about the true state of the world (e.g., a robot doesn't know if a rock is "good" or "bad" until it's close). The challenge is to find a way for the agent to incorporate occasional human suggestions to resolve its own uncertainty more effectively.

Action Suggestions as Bayesian Observations

The core innovation is treating an action suggestion ($o^s$) not as a command to be obeyed, but as a Bayesian observation of the environment.

  1. Modified Belief Update: In a standard POMDP, an agent updates its "belief" (a probability distribution over possible states) based on its actions and environmental observations. The authors add a second update step: if a suggestion is received, the agent calculates the probability of receiving that suggestion given a specific state, $p(o^s | s)$.
  2. Inferring the Suggester’s Intent: Since the agent doesn't have a perfect model of the human, it uses its own optimal policy to estimate what the human is thinking. They propose two models for this:
  3. Scaled Rational: Assumes the suggester follows the agent’s optimal policy with probability $\tau$ and picks a random action otherwise.
  4. Noisy Rational: Uses a Boltzmann (softmax) distribution over the agent's Q-values. This assumes the suggester is more likely to suggest actions with higher expected rewards but might occasionally make small mistakes.

By "thinking" about which states would make the human's suggestion the "right" move, the agent can drastically shift its belief toward the true state of the world. The "aha moment" here is that a suggestion carries implicit information about the suggester's belief. If an Air Traffic Controller suggests a pilot turn left, the pilot realizes there is likely an obstacle to the right that they haven't seen yet. The suggestion changes the pilot's understanding of the world, not just their immediate path.

Experimental Setup

The researchers evaluated their approach on two classic POMDP benchmarks:

  • Tag: An agent must find and "tag" an opponent in a grid. The opponent's position is unobserved unless they are in the same cell.
  • RockSample: A robot navigates a grid to sample rocks. It knows the rock locations but doesn't know which are scientifically valuable ("good") without using a noisy long-range sensor.

These environments are standard in the field because they provide clear metrics for how well an agent manages uncertainty.

Key Results: Efficiency and Robustness

The proposed methods demonstrated significant improvements in both performance and efficiency compared to baseline approaches.

Performance and Efficiency Results
  • Superior Performance: In the "Tag" environment, the proposed methods (Scaled and Noisy Rational) achieved near-perfect rewards, far outperforming a "Normal" agent acting alone.
  • Higher Efficiency: To reach a high score in "Tag," a "Naive" agent (which simply follows the suggestion without updating its belief) required roughly 15 suggestions per run. The proposed methods achieved better results with only 3 suggestions.
  • Robustness to "Bad" Advice: The researchers tested the agents against a "random" suggester. While the Naive agent's performance crashed as suggestions became more random, the proposed Bayesian agents remained robust. Even when 50% of the suggestions were random garbage, the agents still performed better than if they had no help at all.

Limitations and Future Directions

  • The Collaborative Assumption: The model assumes the human and the agent share the exact same objective. If a human is malicious or simply has a different priority (e.g., safety vs. speed), the belief update could lead the agent astray.
  • Offline vs. Online: This work focused on policies generated offline (using the SARSOP algorithm). Extending this to online solvers—where the agent plans its moves in real-time—is a necessary next step.
  • Static Hyperparameters: The "trust" levels ($\tau$ and $\lambda$) were kept constant. A more advanced system might learn to adjust these parameters in real-time based on how helpful the human's suggestions prove to be.

Summary updated

Abstract

The level of autonomy is increasing in systems spanning multiple domains, but these systems still experience failures. One way to mitigate the risk of failures is to integrate human oversight of the autonomous systems and rely on the human to take control when the autonomy fails. In this work, we formulate a method of collaborative decision making through action suggestions that improves action selection without taking control of the system. Our approach uses each suggestion efficiently by incorporating the implicit information shared through suggestions to modify the agent's belief and achieves better performance with fewer suggestions than naively following the suggested actions. We assume collaborative agents share the same objective and communicate through valid actions. By assuming the suggested action is dependent only on the state, we can incorporate the suggested action as an independent observation of the environment. The assumption of a collaborative environment enables us to use the agent's policy to estimate the distribution over action suggestions. We propose two methods that use suggested actions and demonstrate the approach through simulated experiments. The proposed methodology results in increased performance while also being robust to suboptimal suggestions.
BibTeX
@inproceedings{asmar2022collaborative,
  author = {Dylan Asmar and Mykel Kochenderfer},
  title = {{Collaborative Decision Making Using Action Suggestions}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=DylWBluOgqN},
}
Figure