While the tech industry frequently claims that Artificial General Intelligence (AGI) is imminent, recent research from Salesforce suggests that today’s most advanced computer-using agents still struggle with tasks that humans routinely complete over morning coffee. This study takes a scientifically rigorous look at the gap between AI hype and agentic reality through the lens of the popular game Wordle.
The Challenge of Out-of-Distribution Tasks
The AI community is currently focused on "agents"—models capable of navigating the web using a mouse and keyboard. While these agents excel on standard benchmarks like WebArena, these metrics may primarily capture memorization. To test true generalization, researchers chose Wordle. Because the New York Times prohibits bot scraping, Wordle serves as an "out-of-distribution" (OOD) task. Success requires a tight loop of visual perception, logical reasoning, and strategic planning.
Methodology
The study evaluated OpenAI’s Computer-User Agent (CUA), a state-of-the-art model that interacts with GUIs via raw pixel processing. The agent played 200 runs on a live Wordle instance via a Playwright/Chromium setup. To understand the model's internal logic, researchers provided a update_wordle_game_state tool, allowing the agent to "self-annotate" its observations of the board.
Key Insights: Why Agents Fail
Despite its linguistic prowess, the CUA achieved a meager 5.36% success rate. The failure stems from fundamental breakdowns in perception rather than a lack of vocabulary.
The Tokenization Trap and Perceptual Decay
The model's vision is hindered by its architectural design. CUA divides screenshots into 512x512 patches. The Wordle grid often aligns with the boundaries of these patches, creating "blind spots" in the center of the board. Furthermore, as the game progresses and the board fills with color, the model's accuracy plummets.
As shown in the heatmap, accuracy is highest at the edges (positions 1 and 5) and significantly lower in the center (positions 3 and 4). Perceptual accuracy dropped from 42% on the first attempt to just 6% by the fifth.
Optimistic Hallucinations
The model frequently suffered from "positive bias," hallucinating that gray tiles were actually green or yellow. This may be a side effect of Reinforcement Learning from Human Feedback (RLHF), where models are incentivized to appear helpful or successful, leading them to "believe" they are winning despite evidence to the contrary.
Results and Implications
- Success Rate: Only 5.36% (compared to >90% for typical humans).
- Perception Correlation: A strong correlation (0.694) exists between color perception accuracy and game success, confirming that the bottleneck is visual, not linguistic.
- Word Bias: The agent succeeded most with "ARROW" (13.6%) but failed completely on words like "SHEAR."
Conclusion
Achieving true agentic behavior requires moving beyond simple image tokenization toward robust spatial processing and dynamic perception. Until agents can reliably navigate a simple 5x6 grid of colored tiles, the path to AGI remains a significant climb. Future work must focus on generalized reasoning in unseen, multi-step environments rather than memorizing common web workflows.
Summary updated