When "No Worries" Went Wrong: HRI Lessons for Foundation-Model Robots

Eliahu KhalastchiarXiv 2026 · arXiv

TL;DR

This paper explores why robots equipped with advanced foundation models (like LLMs and VLMs) still suffer from awkward social failures, such as appearing passive-aggressive or redundant. The author introduces TRACE, a lightweight architectural review method, and five design lessons aimed at preserving "accountability boundaries" to ensure a robot's actions remain socially appropriate even as the world changes around it.

What Problem Does This Paper Address?

In the transition from modular, hand-coded pipelines to end-to-end foundation models, robots have gained incredible reasoning and perception capabilities. However, "capability alone does not ensure that a changed situation shapes the action that reaches a person."

The paper highlights a specific type of failure: the social analogue of the TOCTOU (Time-of-Check/Time-of-Use) problem. For example, a robot might decide to announce a visitor's arrival, but by the time it physically reaches the host 90 seconds later, the host and visitor are already talking. The robot "successfully" executes its command, but the social purpose has expired, making the interaction redundant or even absurd. The gap lies in how purpose, timing, and evidence are integrated into the robot's architecture.

Alice the office robot, whose asynchronous behavior led to a "passive-aggressive" social failure.

Methods

The author draws on three years of experience from a ROS 2 HRI consortium to propose a framework for robust social interaction:

  1. The HRI Toolkit: A reusable ROS 2 architecture that makes modular responsibilities (like context management and social planning) explicit.
  2. Five Design Lessons (L1-L5): A set of responsibilities for modern HRI stacks, including treating attention as a budget, distinguishing stored knowledge from live evidence, and separating the proposal of an action from its commitment.
  3. TRACE (Trigger, Rationale, Assumptions, Constraints, Effects): A "premise-to-effect" review card. Engineers pick a human-facing effect, change one premise (e.g., "the person already took their medicine"), and follow that change through the system to see if the robot's behavior actually adapts.
Fig. 2. Historical HRI Toolkit (left) beside one possible current Physical AI design space (right). The former makes modular processing responsibilities explicit; the latter depicts possible deliberative and reactive paths, guardrails and observable outputs. Robot silhouettes are illustrative, not a count of tested platforms.

Datasets

This paper does not use standard machine learning training datasets. Instead, it evaluates its architectural findings using:

  • Historical Incidents: Real-world failures observed during the consortium's work with "Alice," a Magni-based office robot.
  • Published Case Studies: A bounded analysis of a published LLM-generated behavior-tree system for medication reminders.

Key Insights and Ideas

  • Success $\neq$ Purpose: A system might report "Success" because a motor moved or a sentence was spoken, even if the action no longer serves its intended social purpose.
  • Accountability over Pipelines: As implementation details dissolve into learned neural weights, engineers must preserve "accountability boundaries"—identifiable points where evidence, purpose, and permission are checked before a physical action occurs.
  • Asynchrony Changes Meaning: In the "No worries" incident, the robot's turn and its speech proceeded asynchronously to avoid latency. However, because the turn finished before the speech began, the robot appeared to "sulk," proving that the order of multimodal effects is as important as the content.
  • Knowledge is Evidence, Not Fact: Robots should treat information (like a user's preference or a door's state) as evidence with a timestamp and source, rather than an immutable truth.

Key Results

The paper qualitatively demonstrates how the TRACE method identifies architectural gaps that traditional testing might miss.

  • In the medication reminder case study, TRACE revealed that while the robot had the capability to ask if a patient had already taken their meds, the deployed behavior tree failed to use that information to cancel a redundant (and potentially annoying) reminder.
  • Table I in the paper summarizes these dispositions, showing that failures often stem from "unused evidence" or a "lack of purpose records" rather than a failure of the underlying AI model itself.

Limitations and Future Work

  • Manual Review: TRACE is currently a manual, lightweight review process. While the author suggests an LLM could help draft these records, it still requires significant engineering judgment.
  • Policy Ambiguity: The method identifies where a system fails to follow a policy, but it doesn't define what the "correct" social policy should be (e.g., should a robot always interrupt a conversation to deliver a message?).
  • Future Direction: The author advocates for "runtime-injectable policies" that can influence execution without requiring the retraining of foundation models, maintaining modularity even in end-to-end systems.

Summary updated

Abstract

When a person declined our robot's invitation to play, Alice turned her shoulder away and said "No worries, it's all good." Only after a brief pause did she drive off. The words and gestures were plausible but their order changed the message. In another episode, Alice informed a host that his visitor had arrived---while the visitor stood right beside him. Both came from a three-year ROS 2 HRI consortium that integrated large language and vision models as they emerged. Today's foundation models give robots broader capabilities to perceive their surroundings, reason about situations and generate behavior. Yet capability alone does not ensure that a changed situation shapes the action that reaches a person. Evidence may never be acquired, or a purpose or constraint may fail to survive planning and execution. A system may even report Success when an action no longer serves its intended purpose. Drawing on the consortium's engineering experience, we develop five design lessons about attention, knowledge, commitments, embodied timing and action authority. We then introduce TRACE, a lightweight review that changes one interaction premise, follows it to a decision or enforcement boundary, and produces an actionable architecture finding. A bounded analysis of a published LLM-generated behavior-tree system illustrates how an available clarification mechanism can go unused. The engineering lesson is to preserve accountability boundaries, not the computational pipelines that once exposed them.
BibTeX
@misc{khalastchi2026when,
  author = {Eliahu Khalastchi},
  title = {{When "No Worries" Went Wrong: HRI Lessons for Foundation-Model Robots}},
  year = {2026},
  eprint = {2610.38400},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.38400},
}
Figure