Abstract
Recognizing failures during task execution and implementing recovery procedures is challenging in robotics. Traditional approaches rely on the availability of extensive data or a tight set of constraints, while more recent approaches leverage large language models (LLMs) to verify task steps and replan accordingly. However, these methods often operate offline, necessitating scene resets and incurring in high costs. This paper introduces Recover, a neuro-symbolic framework for online failure identification and recovery. By integrating ontologies, logical rules, and LLM-based planners, Recover exploits symbolic information to enhance the ability of LLMs to generate recovery plans and also to decrease the associated costs. In order to demonstrate the capabilities of our method in a simulated kitchen environment, we introduce OntoThor, an ontology describing the AI2Thor simulator setting. Empirical evaluation shows that OntoThor's logical rules accurately detect all failures in the analyzed tasks, and that Recover considerably outperforms, for both failure detection and recovery, a baseline method reliant solely on LLMs.
Introduction
Robots operating in dynamic, real-world environments—such as a household kitchen—inevitably encounter execution failures. Whether it is a dropped object, a blocked path, or a violation of a user's specific preference, a robot must be able to identify the error and adapt its plan in real-time. Historically, handling these failures required either massive datasets for every possible contingency or "offline" replanning, which often necessitates resetting the entire environment to a known starting state.
While Large Language Models (LLMs) have emerged as powerful tools for robotic planning, they face significant hurdles when used for failure detection. Purely neural approaches often suffer from hallucinations—suggesting the use of non-existent tools—and high operational costs due to the need for constant status queries. This paper presents Recover, a neuro-symbolic framework designed to bridge these gaps by combining the precision of formal logic with the generative flexibility of LLMs.
The Recover Framework
Recover utilizes a dual-component architecture that separates the task of monitoring from the task of creative problem-solving.
The Symbolic Component: OntoThor
To provide the robot with a structured understanding of its environment, the researchers developed OntoThor, a specialized ontology for the AI2Thor simulator. This ontology maps concepts and relationships within a kitchen setting, including agents, actions, and physical objects. As the robot operates, multi-modal data (audio and video) is converted into symbolic "triples" (e.g., [apple, on-top-of, table]).
Failure detection is handled by hard-coded logical rules written in SPARQL or Prolog. Instead of asking an LLM to guess if a failure occurred, the system checks the state against these rules. For instance, if the robot was holding an object before an action but is no longer holding it afterward—without a deliberate "place" command—the system triggers a DroppingObjFailure alert.
The Neural Component: LLM Planner
When the symbolic layer detects a failure, the framework activates its neural component. The system sends the current state, the ultimate goal, and a specific recovery strategy derived from the ontology to an LLM (GPT-4). The LLM then generates a natural language plan to resolve the issue. To ensure these plans are executable, the system uses sentence embeddings to ground the LLM's suggestions back into a predefined set of robot commands, preventing the model from attempting impossible or non-existent actions.
Experimental Evaluation
The framework was tested within the AI2Thor 3D simulator across 12 distinct tasks. These ranged from simple objectives, such as serving wine (8 steps), to complex procedures like preparing a multi-ingredient salad (33 steps). The system was evaluated against 12 different failure types, including physical accidents and safety violations.
Key Results
- Detection Accuracy: The symbolic sub-goal verifier achieved a 100% success rate in identifying failures. This significantly outperformed the LLM-only baseline, which caught failures only about 50% of the time. Logic-based monitoring proved immune to the context-driven confusion that often plagues neural models.
- Recovery Success: Recover achieved an overall recovery success rate of approximately 70%. Performance was notably higher for simpler tasks (59% total task completion post-recovery) compared to complex tasks (33%).
- Safety and Reliability: The system identified 100% of safety-related issues, such as broken glass on the floor, and successfully recovered from 93% of these critical incidents.
- Cost Efficiency: By utilizing symbolic logic for continuous monitoring and only querying the LLM when a recovery plan was required, the framework drastically reduced computational costs and latency compared to "pure" neural approaches.
Discussion and Insights
A notable finding during evaluation was the "Two-Arm Bias." LLMs frequently suggested recovery plans that required two hands (e.g., holding a container while pouring), despite the robot having only a single gripper. This highlights the tendency of LLMs to rely on human-centric training data rather than the specific constraints of the robotic hardware.
Despite its success, the framework currently faces limitations. It operates in a simulated environment with clean data; transitioning to the "noisy" perception of the real world—where computer vision may misidentify objects—remains a significant challenge. Furthermore, while the LLM is effective for short-term fixes, it still struggles with highly complex, multi-step recovery sequences.
Conclusion
Recover demonstrates that a neuro-symbolic approach provides a more robust and cost-effective solution for robotic failure management than LLMs alone. By leveraging OntoThor for precise failure detection and GPT-4 for flexible replanning, the system achieves high reliability in dynamic environments. Future work will focus on improving the transition to real-world hardware and exploring how the symbolic ontology can facilitate better human-robot communication, allowing robots to explain their failures and recovery strategies in plain language.
Summary updated