TL;DR
This paper introduces a novel framework that fine-tunes a local Vision-Language Model (VLM) to act as a high-level goal planner for Reinforcement Learning (RL) agents. By training on a massive new goal-preference dataset (CCGP-2M) using a token-level optimization method, the authors enable agents to navigate complex, sparse-reward environments like Craftax-Classic with significantly better efficiency and success rates than previous methods.
What Problem Does This Paper Address?
Exploration in "open-world" RL environments is notoriously difficult. In games like Minecraft or Craftax, rewards are sparse and the path to success involves long sequences of specific actions (e.g., you can't mine diamonds without first crafting an iron pickaxe).
While Large Language Models (LLMs) can help by suggesting high-level goals, current methods have significant drawbacks. Some rely on expensive cloud APIs with high latency, while others use LLMs to generate static code or knowledge that doesn't adapt well to the agent's immediate visual state. These general-purpose models aren't "native" to the specific environment, often leading to inconsistent or impractical advice that requires fragile, hand-tuned prompting to fix.
Methods
The authors propose a two-stage framework:
- Offline VLM Fine-Tuning: They start with a base VLM (Qwen3-VL-8B) and fine-tune it on a specialized dataset. The core innovation here is KLTDPO (Kullback-Leibler Token-level Direct Preference Optimization). Unlike standard DPO, which evaluates entire responses, KLTDPO operates at the token level. This allows the model to learn fine-grained preferences—such as the critical importance of goal ordering—more effectively. It also incorporates "distributionally robust reweighting" to focus the model's learning on harder, more informative training samples.
- Online Goal-Guided RL: During gameplay, the fine-tuned VLM acts as a local planner. It processes the agent's visual observation and a text description to generate a prioritized list of goals. A standard RL agent (using Proximal Policy Optimization, or PPO) then uses these goals as a guide to decide its low-level actions.
Datasets
The study introduces CCGP-2M, a multimodal goal-preference dataset for the Craftax-Classic environment.
- Scale: 2 million preference samples covering 43 distinct goals.
- Construction: The authors used powerful models (like DeepSeek-R1 and Claude-Opus) to generate interaction trajectories. They then applied quality-aware selection to keep only the most informative samples.
- Preference Pairs: To teach the model what a "good" plan looks like, they paired successful trajectory data with "negative" examples—plans that were structurally valid but contextually inappropriate (e.g., suggesting a goal the agent had already completed or one it wasn't yet equipped for).
Key Insights and Ideas
- Environment-Specific Adaptation is Crucial: General-purpose LLMs are "jacks of all trades" but masters of none. Fine-tuning a local VLM on environment-specific data allows it to understand the "physics" and logic of the game far better than a prompted cloud model.
- Tokens are the Atoms of Planning: In complex planning, the difference between success and failure can be a single word or the order of two subgoals. Token-level optimization (KLTDPO) captures this nuance, leading to more reliable planners.
- Robustness to "Hard Negatives": By training the model to distinguish between a good plan and a "nearly-good-but-wrong" plan, the authors forced the VLM to develop a deeper understanding of task prerequisites and state-dependent priorities.
Key Results
The proposed method, using the KLTDPO algorithm, outperformed all state-of-the-art baselines on Craftax-Classic:
- Higher Scores: It achieved a mean score of 34.9%, compared to 33.8% for the best baseline (SGRL with DeepSeek-R1).
- Faster Progression: It collected diamonds—one of the game's most difficult achievements—at 3.38 million steps, significantly earlier than the best baseline (3.71 million steps).
- Broad Success: The method ranked first in success rate on 15 out of 22 achievements and second on the remaining 7.
- Training Efficiency: KLTDPO showed faster and more stable convergence in reward margins compared to standard DPO or KLDPO.
Limitations and Future Work
Despite the improvements, the "deepest" achievements in Craftax (like those involving complex combat or rare resource management) still have low success rates across all models, indicating that long-horizon planning is far from solved.
The authors also note that their current evaluation is limited to Craftax-Classic. Future research directions include developing more cost-efficient ways to collect high-quality trajectory data and testing the framework's generalizability across a wider variety of open-world environments.
Summary updated