Anticipate & Act : Integrating LLMs and Classical Planning for Efficient Task Execution in Household Environments

Raghav Arora, Shivam Singh, Karthik Swaminathan, Ahana Datta, Snehasis Banerjee, Brojeshwar Bhowmick, Krishna Murthy Jatavallabhula, Mohan Sridharan, Madhava Krishna

TL;DR

This paper introduces a framework that combines the intuitive "common sense" of Large Language Models (LLMs) with the logical rigor of classical symbolic planners to create more efficient household robots. By using LLMs to anticipate upcoming tasks and a PDDL-based planner to execute them jointly, the system achieves a 31% reduction in execution time compared to traditional one-task-at-a-time approaches.

What Problem Does This Paper Address?

Most current assistive robots are "myopic"—they compute and execute plans for one specific task at a time. If you ask a robot to make coffee and then later ask it to make breakfast, a standard system might go to the fridge twice: once for milk and once for eggs.

Humans don't work this way; we anticipate. We grab the eggs while we’re already at the fridge for the milk. While LLMs are excellent at predicting these high-level task sequences (e.g., "after washing clothes, one usually dries them"), they often struggle with the fine-grained, multi-step physical reasoning required to actually move a robot's joints or navigate a room without errors. Conversely, classical planners are great at low-level logic but require someone to tell them exactly what the goals are. This paper bridges that gap.

Methods

The authors propose a two-stage "Anticipate & Act" pipeline:

  1. High-Level Task Anticipation (The "Brain"): The system uses an LLM (like GPT-4) prompted with a few examples of household routines. Given a partial sequence of tasks (e.g., "User washed the clothes"), the LLM predicts the most likely subsequent tasks (e.g., "Dry clothes," "Iron clothes").
  2. Fine-Grained Planning (The "Hands"): These anticipated tasks are mapped to goal states in PDDL (Planning Domain Definition Language). A symbolic heuristic planner, Fast Downward (FD), then computes a single, optimized sequence of low-level actions (like pickup_obj, move_agent) that satisfies both the current and the anticipated goals simultaneously.
The framework pipeline: (a) LLM predicts tasks, (b) tasks map to PDDL goals, (c) a planner creates a joint action sequence, and (d) execution in simulation.

Datasets

The framework was evaluated in VirtualHome, a realistic 3D simulation environment for household tasks.

  • Task Dataset: The authors created a custom dataset of high-level routines (Cooking, Cleaning, Gardening, etc.), each containing roughly 20 tasks.
  • Domain Complexity: The planning domain included 33 independent actions, 5 rooms, 33 objects, and 19 different receptacles (like fridges or shelves).

Key Insights and Ideas

  • Synergy of Strengths: The core "aha moment" is that LLMs don't need to be perfect planners; they just need to be good goal generators. By letting a classical planner handle the "how," the system avoids the hallucinations and logical lapses common in pure-LLM robotics.
  • Joint Goal Optimization: Planning for multiple goals at once allows the robot to "interleave" actions. As shown in the figure below, the robot can pick up a watering hose while it is already in the pantry to get a lawnmower, saving a redundant trip.
  • Context Awareness: LLMs can adjust their predictions based on natural language context. For example, if a user says, "I have an urgent meeting," the LLM can pivot from suggesting "make breakfast" to "prepare laptop and clean clothes."
Fig. 4: Example plans produced with and without considering anticipated tasks.

Key Results

  • Efficiency Gains: The "Anticipate & Act" approach resulted in a 31% reduction in execution time and a 12% reduction in plan length compared to a myopic baseline.
  • LLM Performance: GPT-4 significantly outperformed PaLM and GPT-3.5 in task anticipation, achieving a near-perfect success ratio when provided with contextual examples.
  • Robustness: The system successfully handled interruptions. When a user changed their mind mid-task, the LLM updated the goal list, and the planner generated a new sequence to "undo" unnecessary actions (like putting vegetables back in the fridge) and start the new priority.

Limitations and Future Work

  • Irreversible Actions: The current system struggles with irreversible physical changes. If the robot cuts a potato for a breakfast that is suddenly cancelled, it cannot "un-cut" the potato.
  • Scalability: While tested in VirtualHome, the authors note the need to explore even more complex domains and eventually move to physical robot hardware.
  • Probabilistic Planning: The current planner assumes deterministic outcomes. Future work could incorporate probabilistic planning to handle the uncertainty of the real world.

Summary updated

Abstract

Assistive agents performing household tasks such as making the bed or cooking breakfast often compute and execute actions that accomplish one task at a time. However, efficiency can be improved by anticipating upcoming tasks and computing an action sequence that jointly achieves these tasks. State-of-the-art methods for task anticipation use data-driven deep networks and Large Language Models (LLMs), but they do so at the level of high-level tasks and/or require many training examples. Our framework leverages the generic knowledge of LLMs through a small number of prompts to perform high-level task anticipation, using the anticipated tasks as goals in a classical planning system to compute a sequence of finer-granularity actions that jointly achieve these goals. We ground and evaluate our framework's abilities in realistic scenarios in the VirtualHome environment and demonstrate a 31% reduction in execution time compared with a system that does not consider upcoming tasks.
BibTeX
@inproceedings{arora2024anticipate,
  author = {Raghav Arora and Shivam Singh and Karthik Swaminathan and Ahana Datta and Snehasis Banerjee and Brojeshwar Bhowmick and Krishna Murthy Jatavallabhula and Mohan Sridharan and Madhava Krishna},
  title = {{Anticipate \& Act : Integrating LLMs and Classical Planning for Efficient Task Execution in Household Environments}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2024},
  url = {https://openreview.net/forum?id=u8uz2G8wzP},
}
Figure