Abstract: Recent work on using natural language to specify commands to robots has grounded that language to LTL. However, mapping natural language task specifications to LTL task specifications using language models require probability distributions over finite vocabulary. Existing state-of-the-art methods have extended this finite vocabulary to include unseen terms from the input sequence to improve output generalization. However, novel out-of-vocabulary atomic propositions cannot be generated using these methods. To overcome this, we introduce an intermediate contextual query representation which can be learned from single positive task specification examples, associating a contextual query with an LTL template. We demonstrate that this intermediate representation allows for generalization over unseen object references, assuming accurate groundings are available. We compare our method of mapping natural language task specifications to intermediate contextual queries against state-of-the-art CopyNet models capable of translating natural language to LTL, by evaluating whether correct LTL for manipulation and navigation task specifications can be output, and show that our method outperforms the CopyNet model on unseen object references. We demonstrate that the grounded LTL our method outputs can be used for planning in a simulated OO-MDP environment. Finally, we discuss some common failure modes encountered when translating natural language task specifications to grounded LTL.
Introduction
To perform complex tasks, robots often translate natural language (NL) into Linear Temporal Logic (LTL), a formal mathematical language that defines goals over time (e.g., "Eventually pick up the mug AND Always stay away from the stairs"). However, current state-of-the-art models struggle with generalization. If a robot is trained to "pick up the red ball" but is told to "pick up the blue cube," it may fail because "blue" and "cube" were not in its training set. Even "CopyNet" models, which can copy words from the input to the output, fail when the required logic involves complex combinations of attributes that are not simple word-for-word copies.
This paper introduces CQ-LTL, a method that allows robots to follow natural language instructions involving objects they have never encountered before. By mapping language to an intermediate "Contextual Query" and then to a "Lifted" (templated) LTL formula, the system separates the structure of a task from the specific objects involved, enabling much better generalization across different environments.
Methodology
The authors propose a three-stage pipeline to decouple task logic from environment specifics. Instead of going straight from natural language to grounded logic, the model first maps the NL to a functional signature called a Contextual Query (CQ). For example, the instruction "Bring the plate to the kitchen" becomes bring("plate", "kitchen").
The core contribution is the CQ-LTL Algorithm, which automatically learns to associate a Contextual Query with a Templated LTL formula using just a single positive example. When a new command arrives, the system extracts the parameters (the new objects), plugs them into the learned template, and generates a "grounded" LTL formula that a robot planner can execute.
The CQ-LTL Framework
A key insight of this work is the use of "Lifted LTL," which employs propositional functions instead of hardcoded objects. This allows the robot to learn the concept of a task once and apply it to any object in any room. Standard CopyNet models fail when the output logic requires generating complex strings like ball_in_small_bucket. CQ-LTL solves this by letting an external grounding function handle the translation of objects, while the model focuses on the task intent.
As shown in the figure above, a complex "ship" task is broken down into its constituent logical parts. By treating the task structure as a template and the objects as parameters, the system can generalize to entirely new object references and attributes not present in the training set.
Evaluation and Results
The approach was evaluated using a simulated Object-Oriented Markov Decision Process (OO-MDP) environment called the "Toy Domain," which supports navigation and object manipulation. The researchers compared CQ-LTL against state-of-the-art CopyNet models on two datasets: a "Seen" dataset with base vocabulary and an "Unseen" dataset with novel object references.
- Manipulation Generalization: On the "Unseen" dataset for manipulation tasks, standard LTL-CopyNet models failed completely (near 0% accuracy) because they could not generate the necessary logic for new objects. In contrast, CQ-LTL maintained over 60% accuracy.
- Navigation Parity: On simpler navigation tasks, CQ-LTL performed as well as state-of-the-art models (~90%+ accuracy), proving that the added flexibility does not hurt performance on standard tasks.
- End-to-End Execution: The authors demonstrated that the LTL generated by their system could be fed directly into a Value Iteration planner to successfully complete tasks in the simulation.
Discussion and Future Work
While CQ-LTL shows significant improvements in generalization, some limitations remain. The system relies on an external grounding function; if a user's reference is ambiguous (e.g., "orange" referring to both a color and a fruit), the system can become confused. Additionally, the pipeline uses Part-of-Speech (POS) tagging to identify parameters. If the tagger misidentifies a word, the resulting contextual query will be malformed.
Future work could explore more complex mappings where one NL term might trigger multiple logical constraints, further refining the robot's ability to understand nuanced human instructions in diverse environments.
Summary updated