Semantic Failure Diagnosis for Robotic Plan Repair

26 concepts19 papers1 proposal

When an autonomous robot fails to execute a task—whether it is missing a peg-in-hole insertion, colliding with a table leg, or failing to hand over an object—traditional control loops encounter a severe information bottleneck. Typically, the low-level execution layer returns a binary error signal: Success: 0.

This binary feedback tells a high-level planner that an action failed, but it completely obscures why. Without causal context, the planner is left in the dark. It cannot distinguish between a minor geometric misalignment, an unreachable kinematic configuration, or a fundamental violation of environmental preconditions. As a result, the robot is forced to either restart the entire plan from scratch or search blindly through alternative strategies.

To overcome this bottleneck, a powerful paradigm has emerged: semantic failure diagnosis. Instead of treating physical execution as a black box that emits binary bits, modern frameworks translate low-level sensorimotor, geometric, or kinematic failures into rich, descriptive semantic explanations. By converting raw physical errors into logical constraints or natural language narratives, these systems provide high-level planners with the exact causal context needed to diagnose mistakes and execute targeted, closed-loop plan repairs.


The Core Mechanism: Translating Physics to Semantics

The fundamental reason semantic failure diagnosis works is that it bridges the abstraction gap between high-level reasoning and low-level execution. High-level planners (such as symbolic solvers or Large Language Models) operate in the currency of discrete concepts, while low-level controllers operate in the currency of continuous physics. Semantic feedback acts as a translator between these two worlds.

[ Low-Level Physical Failure ] 
              │
              ▼ (Kinematic / Tactile / Geometric Analysis)
[ Semantic Explanation / Predicate Violation ]
              │
              ▼ (Re-prompting / Constraint Injection)
[ High-Level Plan Repair ]

Consider a robot attempting to pick up a cup. If the gripper closes on empty air because the robot is positioned too far away, a standard system simply registers a failure. In contrast, the CAPE framework introduced by Shreyas Sundara Raman et al. in [CAPE: Corrective Actions from Precondition Errors using Large Language Models] maps this physical state violation directly to an environmental precondition error. The failure is translated into an explicit natural language description: "I cannot grab the cup because I am not near it."

By feeding this semantic description back into a language model, the model acts as a "commonsense debugger." Because it understands the causal relationship between being near an object and grasping it, it can immediately generate a precise corrective step—such as navigating closer to the counter—rather than abandoning the task.


Bridging Symbolic Logic and Geometric Constraints

This translation is equally vital in traditional Task and Motion Planning (TAMP), where high-level symbolic plans must be checked against low-level geometric feasibility. A symbolic plan might be logically sound (e.g., pick(block) -> move(block) -> place(block)), but physically impossible due to a collision or joint limit along the trajectory.

If the motion solver treats this failure as a simple binary rejection, the symbolic planner must guess an alternative plan at random. To solve this, Joaquim Ortiz de Haro et al. developed a conflict-driven interface in [A Conflict-Driven Interface Between Symbolic Planning and Nonlinear Constraint Solving]. When a physical action proves impossible, the system identifies the exact subset of conflicting geometric constraints and feeds them back to the symbolic layer.

Similarly, the COAST algorithm proposed by Brandon Vu et al. in [COAST: Constraints and Streams for Task and Motion Planning] translates geometric infeasibility into logical PDDL constraints. This targeted feedback forces the high-level symbolic search to prune entire branches of structurally flawed plans and explore fundamentally different topological strategies, resulting in massive planning speedups.


Failure Explanation as an Intermediate Representation

With the integration of Vision-Language Models (VLMs) into robotics, semantic diagnosis has shifted from rigid logical predicates to expressive natural language monologues. A key insight of this research area is that generating an explicit explanation of a failure is a necessary precursor to successful correction.

In [REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction], Zeyi Liu et al. demonstrate that robots that explicitly generate a text summary to analyze why an error occurred (e.g., distinguishing between a physical execution slip and a logical planning flaw) achieve substantially higher recovery success rates than models that attempt to output a correction directly from raw video. The explanation serves as an essential intermediate representation that structures the subsequent planning problem.

[Raw Multisensory Data] ──> [Explicit Text Explanation] ──> [Targeted Recovery Plan]

This concept is pushed further by Jiafei Duan et al. in [AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation], where robotic failure analysis is reframed entirely as a natural language reasoning task rather than a binary classification problem. By training a VLM to output qualitative, linguistic descriptions of errors, the resulting feedback becomes highly actionable for downstream planners and reward-shaping algorithms.

For example, in the LLM³ framework by Shu Wang et al. in [LLM3: Large Language Model-based Task and Motion Planning with Motion Failure Reasoning], low-level motion planning failures are categorized into distinct semantic buckets (such as specific collision types or reachability bounds). This allows a high-level reasoning agent to perform targeted parameter adjustments—like altering a grasp angle or shifting a waypoint—rather than throwing away the entire plan.


Grounding Semantics in Multimodal Sensations

Semantic failure diagnosis does not rely on vision alone; it frequently integrates physical, contact-rich modalities like haptics and force sensing to diagnose hidden environmental properties.

In assembly tasks, visual data is often occluded by the robot's own gripper or the parts themselves. Peter A. Zachares et al. demonstrate in [Interpreting Contact Interactions to Overcome Failure in Robot Assembly Tasks] that interpreting the "tactile feel" (sensorimotor force traces) of a failed insertion allows a robot to simultaneously determine whether the parts are misaligned or if there is a complete geometric mismatch.

To scale this physical intuition to open-world manipulation, Jialei Huang et al. introduced [Tactile-VLA: Unlocking Vision-Language-Action Model's Physical Knowledge for Tactile Generalization]. The system generates an internal natural language monologue to interpret tactile feedback explicitly. By verbalizing physical forces—such as generating the internal reasoning that a slip occurred due to "insufficient downward pressure"—the model can immediately formulate and execute precise physical corrections in its next attempt.


Coordinating Multi-Agent Systems

When tasks scale to multi-robot teams or multi-agent workflows, semantic feedback becomes the primary mechanism for organizational resilience. In complex pipelines, a single failure can cascade across the entire system if not properly contextualized.

In the CONAGENTS framework by Zhengliang Shi et al. in [Learning to Use Tools via Cooperative and Interactive Agents], a dedicated critic agent intercepts raw tool execution errors and refines them into actionable hints for the planning agent. This communication loop prevents rigid pipeline collapses by transforming cryptic system errors into logical alternatives.

At the hardware level, Kehui Liu leverages this closed-loop communication in [COHERENT: Collaboration of Heterogeneous Multi-Robot System with Large Language Models]. In a heterogeneous team comprising drones, quadrupedal dogs, and robotic arms, a binary failure signal from a single robot would paralyze a centralized planner. By requiring individual robots to report descriptive semantic feedback regarding their specific environmental or physical constraints, the centralized LLM "brain" can dynamically reallocate tasks—intelligently pivoting a failed manipulation task from an overloaded robotic arm to an available mobile agent.

By shifting the paradigm from binary error detection to explicit semantic diagnosis, these diverse frameworks allow robots to treat failure not as a fatal stop command, but as a rich source of information for autonomous recovery.

Key Foundational Work

Curly: An AI-based Curling Robot Successfully Competing in the Olympic Discipline of Curling (2018) — contributed 2 concept elements to this direction.
Interpreting Contact Interactions to Overcome Failure in Robot Assembly Tasks (2021) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_semantic_failure_feedback_18bac6a9,
  title = {semantic failure feedback},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semantic-failure-feedback/rdr_18bac6a9385249ad9c596262f074555b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_semantic_failure_feedback_18bac6a9_v1,
  title = {semantic failure feedback},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semantic-failure-feedback/rdr_18bac6a9385249ad9c596262f074555b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake