AGI Is Coming... Right After AI Learns to Play Wordle

Sarath Shekkizhar, Romain CosentinoarXiv 2025 · DOI · arXiv

While the tech industry frequently claims that Artificial General Intelligence (AGI) is imminent, recent research from Salesforce suggests that today’s most advanced computer-using agents still struggle with tasks that humans routinely complete over morning coffee. This study takes a scientifically rigorous look at the gap between AI hype and agentic reality through the lens of the popular game Wordle.

The Challenge of Out-of-Distribution Tasks

The AI community is currently focused on "agents"—models capable of navigating the web using a mouse and keyboard. While these agents excel on standard benchmarks like WebArena, these metrics may primarily capture memorization. To test true generalization, researchers chose Wordle. Because the New York Times prohibits bot scraping, Wordle serves as an "out-of-distribution" (OOD) task. Success requires a tight loop of visual perception, logical reasoning, and strategic planning.

Methodology

The study evaluated OpenAI’s Computer-User Agent (CUA), a state-of-the-art model that interacts with GUIs via raw pixel processing. The agent played 200 runs on a live Wordle instance via a Playwright/Chromium setup. To understand the model's internal logic, researchers provided a update_wordle_game_state tool, allowing the agent to "self-annotate" its observations of the board.

Key Insights: Why Agents Fail

Despite its linguistic prowess, the CUA achieved a meager 5.36% success rate. The failure stems from fundamental breakdowns in perception rather than a lack of vocabulary.

The Tokenization Trap and Perceptual Decay

The model's vision is hindered by its architectural design. CUA divides screenshots into 512x512 patches. The Wordle grid often aligns with the boundaries of these patches, creating "blind spots" in the center of the board. Furthermore, as the game progresses and the board fills with color, the model's accuracy plummets.

Figure 4: Heatmap showing accuracy decay by attempt and position, alongside the grid-patch alignment issue.

As shown in the heatmap, accuracy is highest at the edges (positions 1 and 5) and significantly lower in the center (positions 3 and 4). Perceptual accuracy dropped from 42% on the first attempt to just 6% by the fifth.

Optimistic Hallucinations

The model frequently suffered from "positive bias," hallucinating that gray tiles were actually green or yellow. This may be a side effect of Reinforcement Learning from Human Feedback (RLHF), where models are incentivized to appear helpful or successful, leading them to "believe" they are winning despite evidence to the contrary.

Figure 5: Distribution of error types, highlighting the prevalence of 'Gray to Yellow' and 'Gray to Green' hallucinations.

Results and Implications

  • Success Rate: Only 5.36% (compared to >90% for typical humans).
  • Perception Correlation: A strong correlation (0.694) exists between color perception accuracy and game success, confirming that the bottleneck is visual, not linguistic.
  • Word Bias: The agent succeeded most with "ARROW" (13.6%) but failed completely on words like "SHEAR."

Conclusion

Achieving true agentic behavior requires moving beyond simple image tokenization toward robust spatial processing and dynamic perception. Until agents can reliably navigate a simple 5x6 grid of colored tiles, the path to AGI remains a significant climb. Future work must focus on generalized reasoning in unseen, multi-step environments rather than memorizing common web workflows.

Summary updated

Abstract

This paper investigates multimodal agents, in particular, OpenAI's Computer-User Agent (CUA), trained to control and complete tasks through a standard computer interface, similar to humans. We evaluated the agent's performance on the New York Times Wordle game to elicit model behaviors and identify shortcomings. Our findings revealed a significant discrepancy in the model's ability to recognize colors correctly depending on the context. The model had a $5.36\%$ success rate over several hundred runs across a week of Wordle. Despite the immense enthusiasm surrounding AI agents and their potential to usher in Artificial General Intelligence (AGI), our findings reinforce the fact that even simple tasks present substantial challenges for today's frontier AI models. We conclude with a discussion of the potential underlying causes, implications for future development, and research directions to improve these AI systems.
BibTeX
@misc{shekkizhar2025agi,
  author = {Sarath Shekkizhar and Romain Cosentino},
  title = {{AGI Is Coming... Right After AI Learns to Play Wordle}},
  year = {2025},
  eprint = {2504.15434},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.15434},
}
Figure