Goal-Guided Reinforcement Learning via Fine-Tuning Vision-Language Models

Yajie Qi, Wei Wei, Lin Li, Lijun Zhang, Jianye HaoarXiv 2026 · arXiv

TL;DR

This paper introduces a novel framework that fine-tunes a local Vision-Language Model (VLM) to act as a high-level goal planner for Reinforcement Learning (RL) agents. By training on a massive new goal-preference dataset (CCGP-2M) using a token-level optimization method, the authors enable agents to navigate complex, sparse-reward environments like Craftax-Classic with significantly better efficiency and success rates than previous methods.

What Problem Does This Paper Address?

Exploration in "open-world" RL environments is notoriously difficult. In games like Minecraft or Craftax, rewards are sparse and the path to success involves long sequences of specific actions (e.g., you can't mine diamonds without first crafting an iron pickaxe).

While Large Language Models (LLMs) can help by suggesting high-level goals, current methods have significant drawbacks. Some rely on expensive cloud APIs with high latency, while others use LLMs to generate static code or knowledge that doesn't adapt well to the agent's immediate visual state. These general-purpose models aren't "native" to the specific environment, often leading to inconsistent or impractical advice that requires fragile, hand-tuned prompting to fix.

Comparison of existing LLM-guided RL methods vs. the proposed fine-tuned VLM approach.

Methods

The authors propose a two-stage framework:

  1. Offline VLM Fine-Tuning: They start with a base VLM (Qwen3-VL-8B) and fine-tune it on a specialized dataset. The core innovation here is KLTDPO (Kullback-Leibler Token-level Direct Preference Optimization). Unlike standard DPO, which evaluates entire responses, KLTDPO operates at the token level. This allows the model to learn fine-grained preferences—such as the critical importance of goal ordering—more effectively. It also incorporates "distributionally robust reweighting" to focus the model's learning on harder, more informative training samples.
  2. Online Goal-Guided RL: During gameplay, the fine-tuned VLM acts as a local planner. It processes the agent's visual observation and a text description to generate a prioritized list of goals. A standard RL agent (using Proximal Policy Optimization, or PPO) then uses these goals as a guide to decide its low-level actions.
Overview of the two-stage framework: offline fine-tuning and online goal-guided RL.

Datasets

The study introduces CCGP-2M, a multimodal goal-preference dataset for the Craftax-Classic environment.

  • Scale: 2 million preference samples covering 43 distinct goals.
  • Construction: The authors used powerful models (like DeepSeek-R1 and Claude-Opus) to generate interaction trajectories. They then applied quality-aware selection to keep only the most informative samples.
  • Preference Pairs: To teach the model what a "good" plan looks like, they paired successful trajectory data with "negative" examples—plans that were structurally valid but contextually inappropriate (e.g., suggesting a goal the agent had already completed or one it wasn't yet equipped for).
Distribution of goals in the newly introduced CCGP-2M dataset.

Key Insights and Ideas

  • Environment-Specific Adaptation is Crucial: General-purpose LLMs are "jacks of all trades" but masters of none. Fine-tuning a local VLM on environment-specific data allows it to understand the "physics" and logic of the game far better than a prompted cloud model.
  • Tokens are the Atoms of Planning: In complex planning, the difference between success and failure can be a single word or the order of two subgoals. Token-level optimization (KLTDPO) captures this nuance, leading to more reliable planners.
  • Robustness to "Hard Negatives": By training the model to distinguish between a good plan and a "nearly-good-but-wrong" plan, the authors forced the VLM to develop a deeper understanding of task prerequisites and state-dependent priorities.

Key Results

The proposed method, using the KLTDPO algorithm, outperformed all state-of-the-art baselines on Craftax-Classic:

  • Higher Scores: It achieved a mean score of 34.9%, compared to 33.8% for the best baseline (SGRL with DeepSeek-R1).
  • Faster Progression: It collected diamonds—one of the game's most difficult achievements—at 3.38 million steps, significantly earlier than the best baseline (3.71 million steps).
  • Broad Success: The method ranked first in success rate on 15 out of 22 achievements and second on the remaining 7.
  • Training Efficiency: KLTDPO showed faster and more stable convergence in reward margins compared to standard DPO or KLDPO.
Figure 2. Overview of our framework.

Limitations and Future Work

Despite the improvements, the "deepest" achievements in Craftax (like those involving complex combat or rare resource management) still have low success rates across all models, indicating that long-horizon planning is far from solved.

The authors also note that their current evaluation is limited to Craftax-Classic. Future research directions include developing more cost-efficient ways to collect high-quality trajectory data and testing the framework's generalizability across a wider variety of open-world environments.

Summary updated

Abstract

Exploration remains a major challenge for reinforcement learning (RL) agents in Craftax-style environments due to sparse rewards and long horizons. Large language models (LLMs) can guide exploration by generating high-level goals, but methods using models without explicit task adaptation often rely on carefully designed prompts and may produce inconsistent or contextually inappropriate goals. We introduce CCGP-2M, a multimodal goal-preference dataset constructed from interaction trajectories in Craftax-Classic, and propose a vision-language model (VLM)-guided RL framework that learns environment-specific goal guidance from this dataset. The framework consists of two stages: offline VLM fine-tuning and online goal-guided RL. In the offline stage, we align the VLM with goal preferences using Kullback-Leibler Token-level Direct Preference Optimization (KLTDPO), which combines token-level KL regularization with distributionally robust reweighting. Experiments on Craftax-Classic show that our method collects diamonds earlier than state-of-the-art baselines, achieves the highest score and reward, and attains the highest success rates on most achievements.
BibTeX
@misc{qi2026goal,
  author = {Yajie Qi and Wei Wei and Lin Li and Lijun Zhang and Jianye Hao},
  title = {{Goal-Guided Reinforcement Learning via Fine-Tuning Vision-Language Models}},
  year = {2026},
  eprint = {2611.25742},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25742},
}
Figure