Human-Robot Alignment through Interactivity and Interpretability: Don't Assume a ``Spherical Human''

Matthew Gombolay, Dong Zhou, Xiangyu Shao, Junqiao Wang, Guanghui Sun

TL;DR

This paper argues that for robots to truly integrate into our lives, we must stop treating human teachers as perfect "oracles" and instead account for our inherent biases, suboptimality, and diverse strategies. By combining interactive learning with interpretable models like Decision Trees, the author proposes a framework for "Human-Robot Alignment" where robots and humans engage in a bidirectional dialogue to refine both the robot's behavior and the human's teaching.

What Problem Does This Paper Address?

In physics, a "spherical cow" is a metaphor for oversimplified scientific models. The author posits that roboticists often assume a "spherical human": an expert who provides perfect, noise-free demonstrations and perfectly understands the robot's internal logic.

In reality, humans are messy. We have cognitive biases, our performance fluctuates with workload, and we struggle with the "correspondence problem"—the difficulty of translating our physical actions into a format a robot can execute. Current Reinforcement Learning (RL) often fails in the real world because it requires exhaustive "reward engineering" by experts. Interactive robot learning (learning from demonstrations or language) is a promising alternative, but it currently breaks down because robots can't handle suboptimal human data, and humans can't "see" inside the robot's "black box" to fix mistakes.

Methods

The paper synthesizes several computational techniques developed by the author’s lab to bridge this gap:

  • Learning from Diversity (MSRD & FLAIR): Instead of averaging out different teaching styles, Multi-Style Reward Distillation (MSRD) uses adversarial learning to separate the "common task goal" from "individual strategy." FLAIR extends this to online, lifelong learning.
  • Surpassing the Teacher (SSRR): Self-Supervised Reward Regression (SSRR) allows a robot to perform better than its human demonstrator by mathematically modeling the relationship between noise and performance, then extrapolating toward a "noise-free" ideal.
  • Human Calibration (MIND MELD): This technique creates a personalized model of a specific human's quirks (like reaction delays or over-corrections) to "clean" their data before the robot learns from it.
  • Interpretable Architectures (PNTs): To make robots transparent, the author uses Personalized Neural Trees (PNTs)—differentiable decision trees that are as high-performing as neural networks but can be visualized and "debugged" by non-experts.
  • Bidirectional Feedback (Reciprocal MIND MELD): Rather than just receiving data, the robot provides feedback to the human, coaching them on how to be a better teacher.

Datasets

As this is an "Early Career Track" paper summarizing a body of work, it references several specific application domains and datasets rather than introducing a single new benchmark:

  • Physical Table Tennis: Used to evaluate the FLAIR algorithm for stroke learning.
  • NASA Mars Rover Path Planning: Used to validate DROID (an offline version of MSRD) using data from JPL rover planners.
  • Wildfire Fighting Simulations: Used to test MixTURE for multi-agent coordination.
  • Human-Subject Studies: The paper cites numerous studies involving both younger and older adults (specifically those with Mild Cognitive Impairment) to gather data on trust, workload, and blame attribution.

Key Insights and Ideas

  • Behavior vs. Value Alignment: The author distinguishes between behavior alignment (the "means"—doing the task the way the human does) and value alignment (the "ends"—achieving the outcome the human wants). True alignment requires both.
  • The Interpretability Paradox: Humans actually perform worse at troubleshooting robots when they only watch the robot's physical attempts. They need to see the "why" (the internal logic) to provide helpful corrections.
  • Blame Attribution is Demographic: Research shows that different populations attribute failure differently; some are more likely to blame themselves than the robot, which impacts how they teach.
  • Scaling to Teams: Alignment isn't just 1-on-1. In industrial settings, robots must model human performance as a "moving target" that changes as workers get tired or more experienced.

Key Results

  • FLAIR was shown to outperform user-specific models in both objective performance and subjective user preference in robotic table tennis.
  • SSRR successfully enabled robots to learn policies that exceeded the performance of the suboptimal demonstrators they learned from.
  • MixTURE demonstrated that robots can learn to communicate and act in distributed teams (like wildfire fighting) even when the human teacher has limited visibility.
  • PNTs achieved performance parity with standard deep learning models while remaining human-readable, allowing users to explicitly modify the robot's "logic tree" through a UI.

Limitations and Future Work

  • The "Common Sense" Gap: Robots still lack the inductive biases and "common sense" reasoning that humans take for granted.
  • Sample Efficiency: While interactive learning is better than tabula rasa RL, it still requires too much data/time for many real-world home applications.
  • Foundation Model Ethics: The paper warns that while Large Language Models (LLMs) are a powerful tool for interaction, they can harbor "toxic reasoning" and malignant stereotypes that must be identified and removed before deployment in assistive roles.
  • Abstraction Levels: Future work is needed to find the "Goldilocks" level of abstraction for explanations—too much detail is overwhelming, but too little is unhelpful for debugging.

Summary updated

BibTeX
@inproceedings{gombolay2024human,
  author = {Matthew Gombolay and Dong Zhou and Xiangyu Shao and Junqiao Wang and Guanghui Sun},
  title = {{Human-Robot Alignment through Interactivity and Interpretability: Don't Assume a ``Spherical Human''}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=dLEjaRIviM},
}
Figure