Should artificial agents ask for help in human-robot collaborative problem-solving?

Adrien Bennetot, Vicky Charisi, Natalia Díaz RodríguezarXiv 2020 · arXiv

Abstract

This study investigates the impact of expert assistance on the learning trajectories of reinforcement learning (RL) agents. Using the Towers of Hanoi puzzle as a benchmark, we compare "canonical" scheduled help (turn-taking) with "on-demand" help (asking for assistance based on uncertainty). Our findings reveal an "exploration paradox": while frequent expert intervention provides immediate performance gains, it significantly hinders long-term mastery by restricting the agent's exploration of the state space. Conversely, on-demand help allows agents to leverage expert knowledge without sacrificing the exploratory phase necessary for robust learning.

Introduction

In developmental robotics, researchers often look to human childhood for inspiration on building autonomous, adaptive agents. A hallmark of human intelligence is the ability to seek help when faced with uncertainty. However, in artificial intelligence, the optimal strategy for integrating external help into the learning process remains unclear. This paper addresses whether "canonical" help (scheduled interventions) or "on-demand" help (agent-initiated requests) accelerates learning efficiency in RL agents, using the Towers of Hanoi puzzle as a testbed.

Methods

The researchers employed a standard Q-learning algorithm to solve a 3-disk Towers of Hanoi puzzle, which comprises a state space of 27 configurations. Three primary experimental setups were compared:

  1. Solo Learning: The agent learns entirely through trial and error.
  2. Turn-Taking (Canonical Intervention): An "expert" algorithm, which always knows the optimal move, takes over every $N$ turns (e.g., every 2nd, 3rd, or 4th turn).
  3. Ask-for-Help (On-Demand Intervention): The agent requests help if the highest Q-value for its current state falls below a specific threshold, indicating low confidence.

The primary performance metric was the mean number of moves required to solve the task, averaged over 100 repeated experiments to ensure statistical robustness. The environment provides a clear mathematical optimum of 7 moves for the 3-disk puzzle.

Results and Discussion

The Exploration Paradox

A key finding is the "exploration paradox": constant expert help can act as a "crutch" that impedes long-term learning. In turn-taking scenarios where help is frequent (e.g., every 2 turns), the agent initially outperforms the solo agent. However, because the expert keeps the agent on the optimal path, the agent fails to explore "wrong" states. This lack of exploration results in a convergence time nearly three times longer than that of a solo agent.

Learning curves for different turn-taking frequencies compared to solo performance.

As shown in the figure above, solo Q-learning agents typically reach the optimal 7-move solution after approximately 1,000 training episodes. In contrast, an agent helped every 2 turns requires roughly 3,000 episodes to reach the same level of mastery. The expert's guidance effectively "blinds" the agent to large portions of the state space necessary for building a complete internal model.

Help Frequency and the Power of "Asking"

The frequency of intervention significantly impacts learning. When expert intervention was reduced (e.g., every 4 turns), long-term learning improved, suggesting that learners need "space" to make mistakes.

Furthermore, the "on-demand" agent proved most efficient. By requesting help only when its confidence (Q-values) was low, the agent leveraged expert knowledge to navigate difficult phases without sacrificing the exploratory phase needed for a robust internal map.

Performance of agents using an 'ask-for-help' mechanism versus regular and turn-taking strategies.

The "ask-for-help" mechanism (shown in the figure above) allows the agent to maintain high initial efficiency while converging to the optimum much faster than scheduled turn-taking. This validates hypotheses from human developmental psychology, suggesting that RL agents benefit from help in ways that mirror human children, particularly when help is sought actively.

Conclusion

This research highlights a critical trade-off in collaborative learning: immediate performance gains from expert guidance can come at the cost of long-term mastery. Future work will explore more complex "asking" mechanisms that account for different types of uncertainty and test these hypotheses in larger, more open-ended environments. Additionally, the researchers aim to investigate social dynamics in triadic interactions (e.g., two children and one robot) to further understand how social context influences help-seeking and collaborative problem-solving.

Summary updated

BibTeX
@misc{bennetot2020should,
  author = {Adrien Bennetot and Vicky Charisi and Natalia Díaz Rodríguez},
  title = {{Should artificial agents ask for help in human-robot collaborative problem-solving?}},
  year = {2020},
  eprint = {2006.00882},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2006.00882},
}
Figure