TL;DR
This paper provides a rigorous post-mortem of a Qwen3.5-9B vision-language agent trained to play Pac-Man, identifying why "good" models often fail in practice. By isolating perception, memory, and decision-making, the authors demonstrate that high performance on static benchmarks is a poor predictor of interactive success and that standard RL metrics can mask significant overfitting and catastrophic forgetting.
What Problem Does This Paper Address?
Training Vision-Language Models (VLMs) for long-horizon spatial control—like playing Pac-Man—is notoriously difficult. When an agent fails, developers face a "black box" problem: Did the agent misinterpret the pixels? Did it forget the layout of the maze? Or did it simply choose a lethal action despite having the correct information?
Most research focuses on end-to-end performance, but this paper argues that such an approach hides the root causes of failure. The authors tackle the challenge of diagnosing specific failure modes in the perception-action loop to provide actionable lessons for building more reliable embodied agents.
Methods
The authors evaluate a Qwen3.5-9B based agent using a multi-pronged diagnostic approach:
- Decoupled Evaluation: They separate the agent's components into perception (reading the screen), state maintenance (remembering the board), and decision-making (choosing a direction).
- ASCII Control Study: To isolate decision-making from visual errors, they trained a policy using textual (ASCII) board representations. This removes image transcription noise, allowing them to test if the model can actually "solve" the logic of Pac-Man.
- Curriculum RL: The agent was trained through seven stages of increasing complexity (larger mazes, more ghosts).
- Restart Training (Backplay): They used a technique where the agent starts episodes from states saved just before a previous death, providing a denser learning signal for difficult scenarios.
- Harness Comparisons: They compared "primitive" agents (choosing Up/Down/Left/Right every step) against agents using a "Route-option" harness that selects higher-level goals (e.g., "Collect pellets" or "Avoid ghosts").
Datasets
- Pac-Man Environment: A custom grid-based implementation (up to 25x21 cells) with four ghosts and a 512-step limit.
- Training/Validation Catalog: A frozen set of 32 training mazes and 8 held-out validation mazes (Tier 7).
- D5A Perception Dataset: A reconstructed dataset of Pac-Man screenshots and labels used for Supervised Fine-Tuning (SFT) of the vision component.
Key Insights and Ideas
- The Static-Interactive Gap: A model can achieve near-perfect accuracy on static image transcription (99%+ cell accuracy) yet fail miserably during a game. Interactive errors accumulate; a single misread wall can be "remembered" by the agent, leading it to attempt to walk through walls later.
- Training Wins are a Mirage: High win rates during "restart training" (where the agent only has to clear a few remaining pellets) do not translate to "normal-start" success. As shown in Figure 1, training batch win rates often stay high while actual validation performance fluctuates wildly.
- Catastrophic Forgetting in RL: Continued optimization for the final stage often leads to performance drops in earlier, simpler stages. This "mixed retention" suggests that the agent is overfitting to specific layout patterns rather than learning general spatial reasoning.
- Lethal Decisions on Correct State: Even when the agent "sees" the board perfectly (in the ASCII control), it still makes immediate lethal choices, such as walking into a ghost when an exit is available. This proves that perception is not the only bottleneck.
Key Results
- ASCII Policy Success: By removing visual noise, the agent reached an 80.2% completion rate (77/96 clears) on held-out mazes at its peak.
- Performance Decay: Despite the peak, performance was unstable. The final checkpoint (update 425) dropped to 62.5% (60/96 clears), highlighting the fragility of the learning process.
- Retention Issues: As shown in Table 1, while the agent improved on the final stage (T7), it simultaneously lost ground on several earlier curriculum stages (T3 and T4).
- Perception Accuracy: While the model was 100% exact on a bank of 24 static images, its accuracy dropped when queried periodically during actual gameplay (Table 3).
Limitations and Future Work
- Scope: The study is limited to a single game (Pac-Man) and a single model family (Qwen). The authors acknowledge the lack of multiple training-seed replications.
- Generalization: The results do not establish broad generalization to entirely new task configurations (e.g., different ghost behaviors).
- Future Directions: The authors suggest that future work should focus on distilling explicit search into the model (e.g., a "thinking mode") and developing better credit assignment mechanisms to prevent the agent from reinforcing "lucky" wins that include bad intermediate moves. They advocate for "diagnose-modify-evaluate" loops that target specific failing stages rather than relying on brute-force RL.
Summary updated