TL;DR
This paper introduces RECIPE2PLAN, a novel benchmark for evaluating Large Language Models (LLMs) in complex, real-world multitasking scenarios that include crucial time constraints between actions. They found that while LLMs can handle basic multitasking, they struggle significantly to balance efficiency with feasibility when strict temporal dependencies are introduced, highlighting a need for improved temporal reasoning and global planning capabilities.
What Problem Does This Paper Address?
Current benchmarks for evaluating LLM-based agents often focus on single-task performance or multitasking without strict temporal constraints. This overlooks a critical aspect of real-world applications: the need to manage multiple tasks concurrently while adhering to specific timing requirements between actions. For instance, in cooking or laboratory experiments, certain steps must be performed within precise time windows relative to preceding actions. Failing to do so can lead to task failure or suboptimal outcomes. This paper addresses this gap by proposing a benchmark that specifically challenges LLMs to plan and execute multitask recipes efficiently and feasibly, considering these vital time constraints.
Methods
The authors developed RECIPE2PLAN, a benchmark framework based on real-world cooking recipes. The core of their evaluation involves assessing LLMs' ability to:
- Commonsense Reasoning: Inferring implicit constraints like action dependencies and resource limitations.
- Dynamic Local Planning: Adapting plans as recipe states evolve.
- Strategic Global Planning: Scheduling actions on a timeline to optimize for both efficiency and feasibility, avoiding greedy local optimizations that might violate global time constraints.
They experimented with both open-source (Llama-3.1, Qwen2.5) and closed-source (GPT-40, Gemini-1.5-Pro) LLMs. Their methodology involved prompting models to identify unwritten properties and constraints, followed by a ReAct-style prompting approach for planning and execution. They also explored an "oracle" setting where gold constraints were provided to isolate the impact of LLMs' inherent reasoning capabilities.
Datasets
The primary dataset used is RECIPE2PLAN, a novel benchmark constructed from existing cooking recipes. These recipes were annotated for various properties and constraints, including:
- Action concurrency (autonomous vs. continuous actions)
- Execution interruptibility
- Resource limitations (e.g., oven, microwave, stove)
- Time constraints between specific actions
The dataset consists of 65 diverse recipe combinations designed to test the balance between efficiency and feasibility under time constraints.
Key Insights and Ideas
- Time Constraints are the Bottleneck: The most significant finding is that while LLMs can perform multitasking without explicit time constraints, their performance plummets when these constraints are introduced. This suggests a fundamental weakness in their temporal reasoning and ability to plan globally under such conditions.
- Efficiency vs. Feasibility Trade-off: LLMs struggle to find the sweet spot between maximizing efficiency (completing tasks quickly) and ensuring feasibility (adhering to all constraints). They tend to either prioritize efficiency and violate time constraints or focus heavily on feasibility, sacrificing speed.
- Global Planning is Key: The paper strongly suggests that the primary challenge lies not in commonsense reasoning or local action selection, but in global planning. LLMs need to consider the entire timeline and the interdependencies of all actions to create a truly effective multitask plan.
- Reasoning Models Show Promise but Still Struggle: While reasoning models performed better than non-reasoning models under temporal constraints, they still faced difficulties with action concurrency, indicating that even advanced models have limitations in complex multitasking.
Key Results
- GPT-40's Performance: Even the most capable model, GPT-40, achieved a success rate of only 21.5% in the multitasking scenario with time constraints. Without time constraints, its success rate was much higher (90.8%), demonstrating its ability to plan but not necessarily to do so efficiently and feasibly under temporal pressure.
- Feasibility Priority Improves Success: When prompted to prioritize feasibility over efficiency, LLMs could significantly increase their success rate (e.g., GPT-40's success rate rose from 27.7% to 49.2% with feasibility priority). However, this came at the cost of efficiency.
- Action Dependencies are a Major Source of Invalid Actions: Even with explicit constraints provided, LLMs frequently violated action dependencies, indicating a struggle to correctly sequence tasks.
- Heuristics Outperform LLMs: A simple heuristic baseline consistently outperformed LLMs in both efficiency and feasibility, underscoring the current limitations of LLMs in this domain.
Limitations and Future Work
- Text-Based Environment: The current environment is text-based and does not capture the complexities of physical interaction, exploration, or real-world object manipulation. Future work aims to implement a more realistic, embodied environment.
- Simplified Action Execution: The current setup assumes actions are executed without delay or failure. Future work could explore scenarios with more realistic action execution dynamics.
- Metric Bias: The efficiency metric can be biased by progress rate. While relative efficiency was introduced, the heuristic baseline, while useful, is not guaranteed to be optimal. Future work plans to explore more advanced scheduling algorithms.
- LLM-Modulo Framework: The paper explored the LLM-Modulo framework to improve planning through iterative critiques. While it helped generate more feasible plans, LLMs still struggled to optimize feasible plans for higher efficiency without compromising feasibility.
Summary updated