TL;DR
While Large Language Models (LLMs) excel at generating robot control code via the "Code-as-Policies" paradigm, they often fail in real-world, partially observable environments due to a lack of environmental grounding. To solve this, the authors introduce NESYRO, a neuro-symbolic framework that combines static symbolic verification with interactive runtime validation. When the robot encounters environmental uncertainty, it recursively generates safe exploratory code to gather missing information, drastically improving task success rates.
What Problem Does This Paper Address?
The "Code-as-Policies" paradigm allows LLMs to directly write executable code (like Python scripts) to guide robotic actions, translating high-level natural language instructions into low-level motor skills. This works beautifully in fully observable, pristine simulation environments. However, put that same robot in a messy, dynamic, or partially observable real-world setting, and things quickly fall apart.
If an LLM doesn't know whether a drawer is locked, or lacks precise depth information for a fragile object, it might generate code that blindly attempts an action, leading to catastrophic failures or irreversible damage. Existing frameworks lack a mechanism to say, "I don't know enough about the environment yet; let me safely check first." NESYRO bridges this gap by enabling robots to explicitly reason about uncertainty and verify code correctness before execution.
Methods
NESYRO (Neuro-Symbolic Robot task planning) splits the code generation process into a recursive two-phase pipeline:
- Neuro-Symbolic Code Verification: The LLM generates a task specification alongside the initial policy code. A symbolic verification tool (an SMT solver, specifically Z3) statically checks the code for logical consistency against the specification. If it fails, the solver provides detailed feedback to the LLM for iterative refinement.
- Neuro-Symbolic Code Validation: Once verified, the code is broken down into sequential skills. Each skill is evaluated using a combined Neuro-Symbolic Confidence score ($NeSyConf$), which multiplies Common Sense Confidence (CSC, from a validation LLM) and Logic Confidence (LC, from a PDDL planner).
If the confidence score drops below a set threshold, NESYRO triggers a Safe Probe. The system generates exploratory code to interact with the environment and recover the missing state (e.g., checking if a drawer is locked) without disrupting the overall task. This creates a recursive policy tree where exploratory subroutines ground the parent code before final execution.
Datasets
The authors evaluated NESYRO across both simulation and physical hardware:
- Simulation: They utilized RLBench, a standard benchmark for tabletop robotic manipulation. They tested across four task categories: Object Relocation, Object Interaction, Auxiliary Manipulation, and Long-Horizon tasks.
- Real-World: Experiments were conducted using a physical 7-DoF Franka Emika Research 3 robotic arm equipped with an Intel RealSense D435 RGB-D camera.
To rigorously test partial observability, they introduced four levels of environmental incompleteness: High, Low, Stochastic, and Complete.
Key Insights and Ideas
- Static Verification is Only Half the Battle: Verifying that code is syntactically and logically sound (via SMT solvers) doesn't mean it will work in the real world. Embodied agents require runtime validation that actively probes physical realities.
- Uncertainty as a Subroutine: Instead of failing and completely replanning from scratch when an unexpected state occurs, NESYRO treats exploration as a safe, temporary detour (a probe) that feeds data back into the main program.
- The Neuro-Symbolic Sweet Spot: LLMs provide the common-sense intuition needed to figure out how to explore (e.g., "if it's dark, I should turn on the light"), while symbolic tools (PDDL/SMT) enforce strict logical constraints, preventing the LLM from hallucinating impossible or dangerous actions.
Key Results
NESYRO demonstrated massive improvements over standard LLM code-generation baselines:
- In RLBench simulations, it achieved a 46.2% improvement in task success rate over the state-of-the-art Code-as-Policies baseline.
- In real-world testing under partial observability, NESYRO achieved over 86.8% executability of task-relevant actions.
- It significantly reduced Irreversible Actions (IA)—instances where the robot causes damage or renders a task uncompletable—dropping the total real-world error count from 29 (using the CodeSift baseline) down to just 7.
Limitations and Future Work
While powerful, the current iteration of NESYRO relies on a binary Logic Confidence metric and predefined domain knowledge, which can restrict its adaptability to entirely novel environments. The authors note that future work will focus on integrating probabilistic and temporal reasoning frameworks, such as probabilistic PDDL, to handle more complex real-world nuances. Additionally, they plan to expand the framework so it can validate skills that aren't explicitly predefined in its initial domain knowledge.
Summary updated