Collaborative Decision Making Using Action Suggestions

Dylan M. Asmar, Mykel J. Kochenderfer

Abstract

Autonomous systems often struggle in unmodeled or complex environments. While traditional human-in-the-loop systems rely on manual overrides, this paper proposes a collaborative framework where humans provide action suggestions. By treating these suggestions as independent observations within a Partially Observable Markov Decision Process (POMDP), the agent can update its internal belief state. This approach allows the agent to outperform both unassisted systems and systems that blindly follow human advice, demonstrating increased efficiency and robustness to noisy suggestions.

Introduction

As autonomous systems move from controlled laboratory settings to the messy reality of unstructured environments, their inherent "brittleness" becomes a significant hurdle. When an agent encounters a situation outside its training data or sensor capabilities, it often fails. The standard mitigation strategy—human intervention—typically involves a human taking full control of the system. However, this "manual override" model is demanding for the human operator and can lead to dangerous discontinuities in control.

This paper explores a more nuanced collaboration: the use of action suggestions. In this paradigm, a "suggester" (which could be a human expert or a more sophisticated AI) offers hints about the best course of action. The challenge is to design an agent that can intelligently integrate these hints into its own decision-making process without becoming overly dependent or being led astray by suboptimal advice.

Methodology: Suggestions as Observations

The authors frame the problem of collaborative decision-making using the Partially Observable Markov Decision Process (POMDP) framework. A POMDP agent maintains a "belief state"—a probability distribution over all possible states of the world—which it updates based on observations from its sensors.

The key insight of this work is to treat an action suggestion ($o^s$) not as a command to be obeyed, but as a new type of observation. When the agent receives a suggestion, it performs a Bayesian belief update. This allows the agent to infer information about the hidden state of the environment based on what the suggester recommended. For example, if a suggester recommends moving left, the agent can infer that the suggester believes the goal or a safe path lies in that direction.

Modeling the Suggester

To perform this Bayesian update, the agent needs a model of the suggester's behavior. The paper proposes two distinct models:

  1. Scaled Rational Model: This model assumes the suggester is perfectly rational (i.e., they follow the agent's optimal policy) with a probability $\tau$, and provides a random suggestion with probability $1-\tau$. This parameterizes the agent's trust in the suggester.
  2. Noisy Rational Model: Based on psychological models of human choice, this model assumes the suggester is more likely to suggest actions with higher expected rewards (Q-values). It uses a rationality coefficient ($\lambda$) and a softmax distribution to model the likelihood of different suggestions.

By incorporating these models, the agent can weigh the suggestion against its own prior observations, leading to a more robust and informed belief state.

Experimental Evaluation

The proposed framework was tested on two classic POMDP benchmarks:

  • Tag: An agent must find and tag an opponent in a grid. The opponent's position is unobserved unless the agent is in the same cell. The authors modified this environment to make the opponent's movement more stochastic, increasing the difficulty of tracking.
  • RockSample: A robot explores a grid to sample rocks. While rock locations are known, their value (good or bad) is hidden. The robot can use a noisy, expensive sensor to check a rock, or it can rely on external suggestions to improve its efficiency.

Results

The experiments demonstrated significant improvements across three key metrics: performance, efficiency, and robustness.

Performance and Efficiency

In the Tag environment, an unassisted agent achieved a score of -10.7. Using the "Noisy Rational" model, the agent's performance improved to -1.8, nearly matching the theoretical maximum of -1.7.

In the RockSample task, the focus was on reducing the burden on the suggester. A "Naive" agent that simply followed every suggestion required 15.3 suggestions to complete the task successfully. In contrast, the "Scaled Rational" agent, which used suggestions to update its internal map, achieved the same result with only 6.4 suggestions. This demonstrates that the agent can use suggestions to learn about the environment, eventually requiring less help.

Robustness to Noise

The framework also proved resilient to poor advice. When 50% of the suggestions provided were completely random, the proposed Bayesian agents still outperformed the unassisted "Normal" agent. Meanwhile, the performance of the Naive agent (which obeys all suggestions) plummeted. This highlights the advantage of treating suggestions as information rather than instructions: the agent can effectively "filter" bad advice by comparing it against its own world model.

Conclusion

This research presents a powerful method for human-agent collaboration. By treating action suggestions as observations within a POMDP, agents can benefit from human expertise while maintaining their own autonomy and robustness. Future work will look into real-time learning of suggester reliability and extending the framework to online solvers, where agents can proactively ask for help when uncertainty is high. This move toward truly collaborative systems marks a significant step in making autonomous agents more capable in the real world.

Summary updated

BibTeX
@inproceedings{asmar2022collaborative,
  author = {Dylan M. Asmar and Mykel J. Kochenderfer},
  title = {{Collaborative Decision Making Using Action Suggestions}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=DylWBluOgqN},
}
Figure