GOALNET: Interleaving Neural Goal Predicate Inference with Classical Planning for Generalization in Robot Instruction Following

Jigyasa Gupta, Shreya Sharma, Shreshth Tuli, Rohan Paul, MausamAAAI 2024 · DOI

TL;DR

This paper introduces GOALNET, a novel neuro-symbolic approach that enables robots to understand and execute natural language instructions by interleaving neural subgoal prediction with classical symbolic planning. This method significantly improves task completion rates and generalizes to unseen objects and instructions, addressing limitations of prior imitation learning and pure planning approaches.

What Problem Does This Paper Address?

Robots need to understand high-level instructions given in natural language and translate them into sequences of actions to perform tasks. While imitation learning can learn from human demonstrations, it often struggles to scale with environmental complexity. Symbolic planners, on the other hand, are good at scaling but require explicit goal predicates, not natural language instructions. Previous attempts to bridge this gap by directly predicting final goal predicates from language instructions faced challenges like an exponential hypothesis space for conjunctive goals and susceptibility to irrelevant predicates or side-effects in demonstrations. This paper tackles the problem of enabling robots to learn complex, multi-stage tasks from natural language instructions, with a focus on generalization to novel scenarios and objects.

Methods

GOALNET employs a neuro-symbolic, interleaved approach. Instead of predicting a final, complex goal, it iteratively predicts a single positive and a single negative subgoal predicate at each step. This predicted subgoal is then passed to a classical symbolic planner, which generates a sequence of actions. These actions are executed, updating the robot's world state, and the process repeats.

Key components of GOALNET include:

  • World State Encoder: Encodes the current world state as an object-centric graph, capturing object properties, states, and relations. It leverages ConceptNet embeddings for generalization.
  • Temporal Context Encoder: Uses an LSTM to encode the history of predicted subgoals, providing context for the current prediction.
  • Instruction Conditioned Attention: Aligns the natural language instruction with the world state by attending over objects, allowing the model to focus on task-relevant elements. It uses SentenceBERT for instruction encoding.
  • Sub-goal Decoder: Autoregressively predicts a positive and negative subgoal predicate, using a grammar mask to ensure feasible object-relation combinations.

The authors also introduce GOALNET*, an extension that generalizes to unseen objects by using factored prediction.

Datasets

The paper uses a benchmark dataset from Misra et al. (2015), collected from kitchen and living room environments, featuring multi-stage instructions. This dataset was augmented with data augmentation techniques, including object replacement based on ConceptNet embeddings, to increase the number of training data points and introduce unseen objects.

Key Insights and Ideas

  • Interleaving is Crucial: Iteratively predicting subgoals, planning, and executing actions is more robust than a single end-to-end pipeline. This allows the system to recover from execution errors and earlier subgoal prediction mistakes.
  • Single Subgoal Prediction: Predicting one positive and one negative subgoal at a time significantly reduces the hypothesis space compared to predicting conjunctive goals, making learning more tractable.
  • Neuro-Symbolic Synergy: Combining neural representations for language and world states with classical symbolic planning leverages the strengths of both: generalization from neural models and robust reasoning from symbolic planners.
  • Generalization to Unseen Objects: GOALNET* demonstrates that by decoupling object prediction from the fixed set of training objects, generalization to novel objects is achievable.

Key Results

GOALNET achieves significant improvements over several state-of-the-art baselines, including imitation learning models and other neuro-symbolic approaches. Specifically, it shows a 78% improvement in goal reaching rate compared to several approaches. GOALNET further demonstrates the ability to generalize to scenes with unseen objects and novel paraphrased instructions, achieving up to 1.36 times higher task completion rates*. Ablation studies confirm the importance of each component of GOALNET, with relational information, instance grounding, and temporal context encoding all contributing to performance gains.

Limitations and Future Work

The authors acknowledge that GOALNET struggles in scenarios with multiple plausible subgoals, leading to imperfect scores. Their models currently assume cause-effect modeling via PDDL input, and future work could explore learning symbolic action models directly. They also suggest investigating the modeling of state and goal uncertainty inherent in realistic domains and resolving rare cases where sub-goals might lead to loops.

Summary updated

BibTeX
@inproceedings{gupta2024goalnet,
  author = {Jigyasa Gupta and Shreya Sharma and Shreshth Tuli and Rohan Paul and Mausam},
  title = {{GOALNET: Interleaving Neural Goal Predicate Inference with Classical Planning for Generalization in Robot Instruction Following}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i18.29990},
}
Figure