Mastering PokeGym: Graph-Guided Multimodal Evolution at Test Time

Ruizhi Zhang, Ye Huang, Yuangang Pan, Chuanfu Shen, Zhilin Liu, Ting Xie, Haijun Lei, Lixin DuanarXiv 2026 · arXiv

TL;DR

This paper introduces PokeGym, a novel 3D vision-only benchmark based on Pokémon Legends: Z-A designed to evaluate an agent's ability to learn and adapt across consecutive game episodes without access to internal game states. To conquer this environment, the authors propose G-EvoMAC, a framework that synergistically co-evolves an agent’s visual perception, language strategy, and action macros using a Graph Neural Network (GNN) surrogate to efficiently guide test-time learning.

What Problem Does This Paper Address?

While AI has mastered structured games like chess and Go, open-world 3D games remain a major hurdle for vision-language agents. Most current benchmarks evaluate agents using a fixed configuration or grant them direct access to internal game states. However, real-world deployment requires Test-Time Learning (TTL)—the ability of an agent to adapt and improve its configuration "on the fly" across consecutive episodes of a task based purely on its own experience and visual observations.

Existing self-evolving agent frameworks typically optimize only a single modality in isolation (e.g., tweaking text prompts or generating code for tools). They completely ignore visual perception adjustments and fail to capture the critical cross-modal synergy between seeing, reasoning, and acting in complex 3D environments. Furthermore, evaluating every new agent configuration directly in a 3D game emulator is computationally prohibitive.

Methods

The authors propose Graph-Guided Evolutionary Multimodal Agent Configuration (G-EvoMAC) to jointly optimize three distinct modalities:

  1. Visual Enhancement Pipeline: A per-frame image-processing sequence (cropping, contrast adjustment, sharpening, etc.) that helps the agent focus on task-relevant regions.
  2. Language Strategy: High-level textual guidelines and reasoning prompts distilled by a Large Language Model (LLM) from historical rollouts.
  3. Action Macros: Composite, executable action functions designed to break deadlocks and handle complex navigation.
Figure 3: Overview of the G-EvoMAC framework. (a) The multimodal configuration evolver generates candidate configurations from historical rollouts; (b) the cross-modal synergy performance predictor estimates each candidate’s score and disagreement; (c) the disagreement-aware selector picks the most informative configuration to evaluate, and the observed score is used to update the predictor for the next round.

To avoid the extreme computational cost of running every candidate configuration in the game emulator, G-EvoMAC introduces a Cross-Modal Synergy Performance Predictor. This lightweight GNN surrogate models the candidate configuration as a heterogeneous graph (with vision, language, and action nodes encoded via SigLIP2) and predicts its performance score.

To balance exploration and exploitation, a Disagreement-Aware Selector uses Monte Carlo (MC) Dropout on the GNN to estimate prediction uncertainty. It selects configurations that either show high predicted performance or high network disagreement, ensuring an efficient search through the massive configuration space.

Datasets

The paper introduces and evaluates on two main benchmarks:

  • PokeGym (New): Built on the 3D open-world game Pokémon Legends: Z-A, containing 114 long-horizon tasks split across three categories: Navigation, Interaction, and Target Localization. Agents must act purely from 4 RGB camera views without symbolic maps or state APIs.
  • ALFRED: A standard household embodied-AI benchmark used to test the cross-domain generalization of G-EvoMAC across 255 validation-unseen task instances.
Figure 2: Overview of PokeGym. Top: Task design, showing the proportion of the three task categories along with example visual trajectories. Bottom: The environment interface, where the agent perceives the game through visual observations and interacts with the emulator via high-level actions.

Key Insights and Ideas

  • Visual Grounding is the Core Bottleneck: Ablation studies revealed that removing the visual enhancement pipeline caused the most drastic performance drop (from 60.18% to 39.82% success rate). Enhancing raw visual inputs is critical for 3D spatial reasoning.
  • Cross-Modal Attention Alignment: Attention weight analysis within the GNN showed that the strongest links formed between Strategy $\rightarrow$ Action and Vision $\rightarrow$ Action. This confirms that effective action macros must be grounded simultaneously in language instructions and visual cues.
  • Uncertainty as a Search Guide: Pure exploration or pure exploitation strategies underperform. Using the GNN's internal disagreement (via MC-Dropout) provides a highly reliable signal for balancing the exploration-exploitation trade-off during test-time evolution.

Key Results

  • PokeGym Performance: G-EvoMAC achieved a 60.18% average success rate, outperforming the strongest baseline (such as Voyager and EvoTest) by over 11 percentage points.
  • Efficiency: It significantly reduced the Ineffective Move Rate (IMR)—steps resulting in no spatial displacement—to just 5.55%.
  • Predictor Accuracy: The GNN surrogate proved highly accurate, with predicted scores strongly correlating with true scores (Pearson $r = 0.991$).
Calibration curves of the GNN predictor
  • Cross-Domain Generalization: When evaluated on ALFRED using a Qwen3.6-35B backbone, G-EvoMAC achieved a 15.80% average Goal-Condition Success (GCS), notably outperforming zero-shot and single-modality evolution baselines.

Limitations and Future Work

Despite its success, G-EvoMAC has a few notable limitations:

  • API and Token Costs: Each run consumes roughly 163k input tokens, costing about \$0.560 per run with GPT-5.4, which could limit large-scale hyperparameter tuning for researchers on a budget.
  • Inter-Episode vs. Intra-Episode Adaptation: The framework currently evolves configurations between full episodes. A natural next step is developing lightweight, online update rules that allow the agent to adapt its visual and action configurations within a single playthrough.
  • Lack of Explicit 3D Understanding: Incorporating explicit 3D scene representations—like depth estimation or semantic mapping—could further resolve remaining spatial-reasoning failures.

Summary updated

Abstract

While artificial intelligence has mastered structured games like chess and Go, vision-language agents still struggle in visually-driven 3D games without access to game states. Existing game environments typically evaluate a fixed agent configuration, rather than an agent's ability to improve its configuration across consecutive episodes of the same task---a paradigm known as test-time learning (TTL). Furthermore, current TTL methods typically optimize single modalities---such as text prompts or actions---in isolation, ignoring the synergy between perception, reasoning, and control. To bridge these gaps, we first introduce \textbf{PokeGym}, a long-horizon benchmark built upon the 3D open-world game Pokémon Legends: Z-A, where agents act from visual observations without access to game states, designed to evaluate an agent's ability to learn and adapt across consecutive episodes of the task. To tackle this challenging environment, we propose Graph-Guided Evolutionary Multimodal Agent Configuration (\textbf{G-EvoMAC}), a graph-guided framework that jointly optimizes visual perception, strategy, and action set synergistically. Extensive experiments show that G-EvoMAC achieves a 60.18\% average success rate on PokeGym, outperforming the strongest baseline by over 11 percentage points, validating the power of cross-modal co-evolution.
BibTeX
@misc{zhang2026mastering,
  author = {Ruizhi Zhang and Ye Huang and Yuangang Pan and Chuanfu Shen and Zhilin Liu and Ting Xie and Haijun Lei and Lixin Duan},
  title = {{Mastering PokeGym: Graph-Guided Multimodal Evolution at Test Time}},
  year = {2026},
  eprint = {2604.08340},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.08340},
}
Figure