Generalizing to New Domains by Mapping Natural Language to Lifted LTL

Eric Hsiung, Hiloni Mehta, Junchi Chu, Jason Xinyu Liu, Roma Patel, Stefanie Tellex, George Konidaris

TL;DR

This paper introduces CQ-LTL, a method that allows robots to follow natural language instructions involving objects they have never encountered before. By mapping language to an intermediate "Contextual Query" and then to a "Lifted" (templated) Linear Temporal Logic (LTL) formula, the system separates the structure of a task from the specific objects involved, enabling much better generalization across different environments.

What Problem Does This Paper Address?

To perform complex tasks, robots often translate natural language (NL) into Linear Temporal Logic (LTL), a formal mathematical language that defines goals over time (e.g., "Eventually pick up the mug AND Always stay away from the stairs").

Current state-of-the-art models (like Seq2Seq) struggle with two main issues:

  1. Fixed Vocabularies: If a robot is trained to "pick up the red ball" but is told to "pick up the blue cube," it may fail because "blue" and "cube" weren't in its training set. Even "CopyNet" models, which can copy words from the input to the output, fail when the required logic involves complex combinations of attributes (like ObjectIsColor(obj, blue)) that aren't simple word-for-word copies.
  2. Domain Locking: Most models output "grounded" LTL, which is tied to specific objects in a specific room. This makes it impossible to transfer the "logic" of a task (like "shipping an item") from a warehouse to a kitchen without retraining the entire model.

Methods

The authors propose a three-stage pipeline to decouple task logic from environment specifics:

  1. Contextual Query (CQ): Instead of going straight to logic, the model first maps the NL to a functional signature. For example, "Bring the plate to the kitchen" becomes bring("plate", "kitchen").
  2. Lifted LTL (Templates): The researchers define "Lifted LTL," which uses propositional functions (templates) instead of hardcoded objects. A "ship" task template might look like: Eventually(ObjectInBox(item, box) AND Eventually(BoxInRoom(box, room))).
  3. The CQ-LTL Algorithm: This is the core contribution. It automatically learns to associate a Contextual Query (the function) with a Templated LTL formula using just a single positive example.

When a new command arrives, the system extracts the parameters (the new objects), plugs them into the learned template, and generates a "grounded" LTL formula that a robot planner can actually execute.

Datasets

The authors evaluated their approach using a simulated OO-MDP (Object-Oriented Markov Decision Process) environment called the "Toy Domain," which supports both navigation and object manipulation (picking, placing, and moving containers).

  • Seen Dataset (S): Contains thousands of examples using a base vocabulary of objects and locations.
  • Unseen Dataset (U): Contains tasks involving entirely new object references and attributes not present in the training set.
  • Task Classes: Included move_to, pickup, pickup_colored, and ship (a complex multi-step task).

Key Insights and Ideas

  • Separation of Concerns: By treating the task structure as a template and the objects as parameters, the robot can learn the concept of "shipping" once and apply it to any object in any room.
  • Overcoming the "Copy" Limit: Standard CopyNet models fail when the output logic requires generating complex strings like ball_in_small_bucket. CQ-LTL solves this by letting an external grounding function handle the "translation" of objects, while the model focuses on the "intent."
  • One-Shot Mapping: The mapping between a query like bring(x, y) and its LTL structure is so consistent that the system can learn the relationship from a single example, provided the parameters are distinct.

Key Results

  • Manipulation Generalization: On the "Unseen" dataset for manipulation tasks, standard LTL-CopyNet models failed completely (near 0% accuracy) because they couldn't generate the necessary logic for new objects. In contrast, CQ-LTL maintained over 60% accuracy.
  • Navigation Parity: On simpler navigation tasks (where nouns often map 1:1 to locations), CQ-LTL performed as well as state-of-the-art models (~90%+ accuracy), proving that the added flexibility doesn't hurt performance on standard tasks.
  • End-to-End Execution: The authors demonstrated that the LTL generated by their system could be fed directly into a Value Iteration planner to successfully complete tasks in the simulation.

Limitations and Future Work

  • Grounding Ambiguity: The system relies on an external grounding function. If a user says "pick up the orange," and "orange" refers to both a color and a fruit in the environment, the system can get confused.
  • POS Tagging Errors: The pipeline uses Part-of-Speech (POS) tagging to identify parameters. If the tagger misidentifies a verb (like "pickup") as a noun, the contextual query will be malformed.
  • Parameter Matching: The current algorithm assumes parameters in the NL map uniquely to parameters in the LTL. Future work could explore more complex mappings where one NL term might trigger multiple logical constraints.

Summary updated

Abstract

Recent work on using natural language to specify commands to robots has grounded that language to LTL. However, mapping natural language task specifications to LTL task specifications using language models require probability distributions over finite vocabulary. Existing state-of-the-art methods have extended this finite vocabulary to include unseen terms from the input sequence to improve output generalization. However, novel out-of-vocabulary atomic propositions cannot be generated using these methods. To overcome this, we introduce an intermediate contextual query representation which can be learned from single positive task specification examples, associating a contextual query with an LTL template. We demonstrate that this intermediate representation allows for generalization over unseen object references, assuming accurate groundings are available. We compare our method of mapping natural language task specifications to intermediate contextual queries against state-of-the-art CopyNet models capable of translating natural language to LTL, by evaluating whether correct LTL for manipulation and navigation task specifications can be output, and show that our method outperforms the CopyNet model on unseen object references. We demonstrate that the grounded LTL our method outputs can be used for planning in a simulated OO-MDP environment. Finally, we discuss some common failure modes encountered when translating natural language task specifications to grounded LTL.
BibTeX
@inproceedings{hsiung2022generalizing,
  author = {Eric Hsiung and Hiloni Mehta and Junchi Chu and Jason Xinyu Liu and Roma Patel and Stefanie Tellex and George Konidaris},
  title = {{Generalizing to New Domains by Mapping Natural Language to Lifted LTL}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2022},
  url = {https://openreview.net/forum?id=szdRSikt6C},
}

Loading related content...

Figure