TL;DR
This paper introduces PokeGym, a novel 3D vision-only benchmark based on Pokémon Legends: Z-A designed to evaluate an agent's ability to learn and adapt across consecutive game episodes without access to internal game states. To conquer this environment, the authors propose G-EvoMAC, a framework that synergistically co-evolves an agent’s visual perception, language strategy, and action macros using a Graph Neural Network (GNN) surrogate to efficiently guide test-time learning.
What Problem Does This Paper Address?
While AI has mastered structured games like chess and Go, open-world 3D games remain a major hurdle for vision-language agents. Most current benchmarks evaluate agents using a fixed configuration or grant them direct access to internal game states. However, real-world deployment requires Test-Time Learning (TTL)—the ability of an agent to adapt and improve its configuration "on the fly" across consecutive episodes of a task based purely on its own experience and visual observations.
Existing self-evolving agent frameworks typically optimize only a single modality in isolation (e.g., tweaking text prompts or generating code for tools). They completely ignore visual perception adjustments and fail to capture the critical cross-modal synergy between seeing, reasoning, and acting in complex 3D environments. Furthermore, evaluating every new agent configuration directly in a 3D game emulator is computationally prohibitive.
Methods
The authors propose Graph-Guided Evolutionary Multimodal Agent Configuration (G-EvoMAC) to jointly optimize three distinct modalities:
- Visual Enhancement Pipeline: A per-frame image-processing sequence (cropping, contrast adjustment, sharpening, etc.) that helps the agent focus on task-relevant regions.
- Language Strategy: High-level textual guidelines and reasoning prompts distilled by a Large Language Model (LLM) from historical rollouts.
- Action Macros: Composite, executable action functions designed to break deadlocks and handle complex navigation.
To avoid the extreme computational cost of running every candidate configuration in the game emulator, G-EvoMAC introduces a Cross-Modal Synergy Performance Predictor. This lightweight GNN surrogate models the candidate configuration as a heterogeneous graph (with vision, language, and action nodes encoded via SigLIP2) and predicts its performance score.
To balance exploration and exploitation, a Disagreement-Aware Selector uses Monte Carlo (MC) Dropout on the GNN to estimate prediction uncertainty. It selects configurations that either show high predicted performance or high network disagreement, ensuring an efficient search through the massive configuration space.
Datasets
The paper introduces and evaluates on two main benchmarks:
- PokeGym (New): Built on the 3D open-world game Pokémon Legends: Z-A, containing 114 long-horizon tasks split across three categories: Navigation, Interaction, and Target Localization. Agents must act purely from 4 RGB camera views without symbolic maps or state APIs.
- ALFRED: A standard household embodied-AI benchmark used to test the cross-domain generalization of G-EvoMAC across 255 validation-unseen task instances.
Key Insights and Ideas
- Visual Grounding is the Core Bottleneck: Ablation studies revealed that removing the visual enhancement pipeline caused the most drastic performance drop (from 60.18% to 39.82% success rate). Enhancing raw visual inputs is critical for 3D spatial reasoning.
- Cross-Modal Attention Alignment: Attention weight analysis within the GNN showed that the strongest links formed between Strategy $\rightarrow$ Action and Vision $\rightarrow$ Action. This confirms that effective action macros must be grounded simultaneously in language instructions and visual cues.
- Uncertainty as a Search Guide: Pure exploration or pure exploitation strategies underperform. Using the GNN's internal disagreement (via MC-Dropout) provides a highly reliable signal for balancing the exploration-exploitation trade-off during test-time evolution.
Key Results
- PokeGym Performance: G-EvoMAC achieved a 60.18% average success rate, outperforming the strongest baseline (such as Voyager and EvoTest) by over 11 percentage points.
- Efficiency: It significantly reduced the Ineffective Move Rate (IMR)—steps resulting in no spatial displacement—to just 5.55%.
- Predictor Accuracy: The GNN surrogate proved highly accurate, with predicted scores strongly correlating with true scores (Pearson $r = 0.991$).
- Cross-Domain Generalization: When evaluated on ALFRED using a Qwen3.6-35B backbone, G-EvoMAC achieved a 15.80% average Goal-Condition Success (GCS), notably outperforming zero-shot and single-modality evolution baselines.
Limitations and Future Work
Despite its success, G-EvoMAC has a few notable limitations:
- API and Token Costs: Each run consumes roughly 163k input tokens, costing about \$0.560 per run with GPT-5.4, which could limit large-scale hyperparameter tuning for researchers on a budget.
- Inter-Episode vs. Intra-Episode Adaptation: The framework currently evolves configurations between full episodes. A natural next step is developing lightweight, online update rules that allow the agent to adapt its visual and action configurations within a single playthrough.
- Lack of Explicit 3D Understanding: Incorporating explicit 3D scene representations—like depth estimation or semantic mapping—could further resolve remaining spatial-reasoning failures.
Summary updated