TravelPlanner: A Benchmark for Real-World Planning with Language Agents

Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, Yu Su

Abstract

Planning has been part of the core pursuit for artificial intelligence since its conception, but earlier AI agents mostly focused on constrained settings because many of the cognitive substrates necessary for human-level planning have been lacking. Recently, language agents powered by large language models (LLMs) have shown interesting capabilities such as tool use and reasoning. Are these language agents capable of planning in more complex settings that are out of the reach of prior AI agents? To advance this investigation, we propose TravelPlanner, a new planning benchmark that focuses on travel planning, a common real-world planning scenario. It provides a rich sandbox environment, various tools for accessing nearly four million data records, and 1,225 meticulously curated planning intents and reference plans. Comprehensive evaluations show that the current language agents are not yet capable of handling such complex planning tasks—even GPT-4 only achieves a success rate of 0.6%. Language agents struggle to stay on task, use the right tools to collect information, or keep track of multiple constraints. However, we note that the mere possibility for language agents to tackle such a complex problem is in itself non-trivial progress. TravelPlanner provides a challenging yet meaningful testbed for future language agents.

Introduction

While Large Language Models (LLMs) have demonstrated remarkable abilities in isolated reasoning and tool-use tasks, their capacity for long-horizon planning remains largely untested in complex, real-world environments. Travel planning represents a quintessential challenge: it requires an agent to synthesize dozens of interdependent decisions over multiple days while adhering to a strict set of constraints. These include hard constraints (e.g., budget, specific dates, pet-friendly accommodations) and implicit commonsense constraints (e.g., logical city routes and diverse dining options). TravelPlanner is designed to evaluate whether current agents possess the cognitive substrates required to navigate these messy, multi-layered requirements.

The TravelPlanner Framework

To rigorously test these capabilities, the researchers developed a comprehensive sandbox environment. The framework operates by taking a user's natural language intent and requiring an agent to analyze constraints, collect information via specialized tools, and deliver a coherent itinerary.

The TravelPlanner framework: from user intent and constraint analysis to information collection and final itinerary delivery.

The benchmark offers two primary evaluation modes:

  1. Two-Stage Mode (Tool-Use + Planning): The agent must proactively use six search tools (FlightSearch, AccommodationSearch, etc.) to gather data from a database of 4 million records, store relevant notes, and then generate a final plan.
  2. Sole-Planning Mode: To isolate planning logic from information retrieval, the agent is provided with all necessary data upfront and must assemble a valid itinerary.

Datasets and Constraints

The benchmark consists of nearly 4 million data records crawled from the web, ensuring a "closed-world" evaluation that prevents models from relying on external training data or hallucinations. The test set includes 1,225 curated queries categorized into Easy, Medium, and Hard levels based on trip duration (3, 5, or 7 days) and the density of constraints. Human-annotated reference plans ensure that every query has at least one feasible solution.

Results and Failure Analysis

The quantitative results provide a sobering reality check for the state of AI agency. In the full two-stage task, GPT-4-Turbo was the only model to achieve any successful plans, yet it reached a success rate of only 0.6%. All other tested models, including Gemini Pro and Mixtral-7x8B, failed to produce a single completely valid plan (0% success rate).

Even in the Sole-Planning mode, where the search burden was removed, GPT-4-Turbo only reached a 4.4% success rate. The failures are not merely due to a lack of information but a fundamental inability to manage global constraints. As the number of constraints increases, performance drops precipitously.

Failure modes of various LLMs in the TravelPlanner environment, highlighting issues like exceeding step limits and invalid action loops.

Analysis of failure modes reveals that agents often get trapped in "dead loops," repeating failed searches, or exceed the maximum step limit without reaching a conclusion. Furthermore, there is a significant "Reasoning vs. Action" gap: agents frequently identify a constraint violation (e.g., being over budget) in their internal thought process but fail to take the corrective action (e.g., selecting a cheaper hotel) in the subsequent step.

Conclusion

TravelPlanner highlights a significant gap between the perceived reasoning capabilities of LLMs and their actual performance in complex planning scenarios. Current agents struggle with long-horizon consistency and the simultaneous satisfaction of multiple constraints. While the 0.6% success rate of GPT-4-Turbo underscores the difficulty of the task, the fact that agents can attempt such a complex problem marks a new frontier in AI research. Future work must focus on developing more sophisticated world models and planning architectures that allow for global optimization and backtracking rather than simple linear, auto-regressive execution.

Summary updated

Abstract

Planning has been part of the core pursuit for artificial intelligence since its conception, but earlier AI agents mostly focused on constrained settings because many of the cognitive substrates necessary for human-level planning have been lacking. Recently, language agents powered by large language models (LLMs) have shown interesting capabilities such as tool use and reasoning. Are these language agents capable of planning in more complex settings that are out of the reach of prior AI agents? To advance this investigation, we propose TravelPlanner, a new planning benchmark that focuses on travel planning, a common real-world planning scenario. It provides a rich sandbox environment, various tools for accessing nearly four million data records, and 1,225 meticulously curated planning intents and reference plans. Comprehensive evaluations show that the current language agents are not yet capable of handling such complex planning tasks-even GPT-4 only achieves a success rate of 0.6%. Language agents struggle to stay on task, use the right tools to collect information, or keep track of multiple constraints. However, we note that the mere possibility for language agents to tackle such a complex problem is in itself non-trivial progress. TravelPlanner provides a challenging yet meaningful testbed for future language agents.
BibTeX
@inproceedings{xie2024travelplanner,
  author = {Jian Xie and Kai Zhang and Jiangjie Chen and Tinghui Zhu and Renze Lou and Yuandong Tian and Yanghua Xiao and Yu Su},
  title = {{TravelPlanner: A Benchmark for Real-World Planning with Language Agents}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=l5XQzNkAOe},
}
Figure