Language-based Trial and Error Falls Behind in the Era of Experience

Haoyu Wang, Guozheng Ma, Shugang Cui, Yilun Kong, Haotian Luo, Li Shen, Mengya Gao, Yichao Wu, Xiaogang Wang, Dacheng Tao

TL;DR

SCOUT (Sub-Scale Collaboration On Unseen Tasks) is a novel framework that overcomes the high cost of LLM exploration by using lightweight "scout" models (like tiny MLPs) to map out new environments. By distilling these efficient trial-and-error trajectories into a larger LLM, the framework achieves state-of-the-art performance on symbolic and spatial tasks—outperforming models like Gemini-2.5-Pro—while slashing GPU training costs by 60%.

What Problem Does This Paper Address?

Large Language Models (LLMs) are brilliant at linguistic reasoning but surprisingly brittle when dropped into "unseen" non-linguistic environments, such as symbolic puzzles (Sudoku) or spatial challenges (Rubik’s Cube).

The authors argue that the bottleneck isn't necessarily a lack of "intelligence," but rather the prohibitive cost of exploration. LLMs are parameter-heavy giants; asking them to perform the thousands of trial-and-error steps needed to learn the "physics" of a new environment is computationally ruinous. Furthermore, there is a "dimension mismatch": LLMs operate in a high-dimensional semantic space (thousands of tokens), while most symbolic tasks require only a few discrete actions. Forcing a billion-parameter model to "think" through every failed attempt at a 4x4 Sudoku grid is like using a supercomputer to flip a light switch.

Methods

The authors propose SCOUT, a three-stage pipeline that decouples exploration from exploitation:

  1. Exploration Stage: Instead of the LLM, the authors deploy "scouts"—tiny neural networks (MLPs or CNNs) that are roughly $10^5$ times smaller than the LLM. These scouts use classic Reinforcement Learning (DQN or PPO) to rapidly interact with the environment. Because they are so small, they can run millions of trials on a CPU at negligible cost.
  2. Distillation Stage: The successful trajectories from the scouts are converted into natural language dialogues using a "Textualizer." The LLM then undergoes Supervised Fine-Tuning (SFT) on these expert paths. This "warms up" the LLM, teaching it the basic rules and dynamics of the task without it having to fail thousands of times itself.
  3. Evolving Stage: Finally, the LLM performs multi-turn Reinforcement Learning (PPO) to refine its reasoning. This stage "activates" the LLM’s latent knowledge, allowing it to move beyond mere imitation of the scout to high-level strategic planning.

Datasets

The framework was evaluated on six symbolic and spatial tasks, chosen for their high "state perplexity" (meaning they are truly out-of-distribution for standard LLMs):

  • Bandit: A classic reward-maximization sanity check.
  • FrozenLake: Grid navigation with "slippery" (stochastic) dynamics.
  • Sokoban: A complex box-pushing planning puzzle.
  • Sudoku: Pure symbolic constraint satisfaction.
  • 2048: A long-horizon task requiring over 800 turns to succeed.
  • Rubik’s Cube (2x2): A spatial intelligence task requiring 3D mental simulation.

Key Insights and Ideas

  • Decoupling is Key: The most significant "aha moment" is that we shouldn't make LLMs do the "dirty work" of initial exploration. Small models are better at the high-frequency trial-and-error needed to learn "physics," while LLMs are better at the high-level reasoning needed to master the task.
  • Latent Activation: In tasks like Sudoku, the Distillation stage (SFT) only yielded a 0.29 success rate. However, it provided the "latent structural understanding" that allowed subsequent RL to rapidly "activate" the model, jumping to a 0.97 success rate.
  • The "Bitter Lesson" Applied: The paper embraces the idea that computation (search and learning) eventually beats human-engineered priors. By making that computation 60% cheaper, they make the "Bitter Lesson" more accessible.
  • Implicit to Explicit Modeling: During the Evolving stage, the LLM naturally began to fill in blank <think> tags with actual analysis, transitioning from just "doing" the task to "reasoning" about it.

Key Results

  • Superior Performance: A Qwen2.5-3B-Instruct model equipped with SCOUT achieved an average score of 0.86 across all tasks. For comparison, the proprietary Gemini-2.5-Pro scored only 0.60, and DeepSeek-V3 scored 0.44.
  • Efficiency Gains: SCOUT achieved these results while saving 60% of GPU hours compared to training an LLM with direct Reinforcement Learning.
  • Multi-task Stability: The framework showed remarkable resistance to "catastrophic forgetting." When trained sequentially on all five tasks, the model maintained a 0.91 average score, even seeing "positive transfer" where learning one task helped it reason better in another.

Limitations and Future Work

The authors acknowledge a few key limitations:

  • Model Scale: Due to resource constraints, they did not validate the framework on "frontier-scale" models (e.g., 70B+ parameters), though they expect the efficiency gains to be even more pronounced there.
  • RL Stability: Like most multi-turn RL approaches, they observed some performance degradation after many training steps, suggesting that stabilizing long-term RL training remains an open challenge.
  • Algorithm Variety: While they used PPO and DQN, they note that newer algorithms like GRPO might offer further improvements.

Summary updated

Abstract

While Large Language Models (LLMs) excel in language-based agentic tasks, their applicability to unseen, nonlinguistic environments (e.g., symbolic or spatial tasks) remains limited. Previous work attributes this performance gap to the mismatch between the pretraining distribution and the testing distribution. In this work, we demonstrate the primary bottleneck is the prohibitive cost of exploration: mastering these tasks requires extensive trial-and-error, which is computationally unsustainable for parameter-heavy LLMs operating in a high dimensional semantic space. To address this, we propose SCOUT (Sub-Scale Collaboration On Unseen Tasks), a novel framework that decouples exploration from exploitation. We employ lightweight "scouts" (e.g., small MLPs) to probe environmental dynamics at a speed and scale far exceeding LLMs. The collected trajectories are utilized to bootstrap the LLM via Supervised Fine-Tuning (SFT), followed by multi-turn Reinforcement Learning (RL) to activate its latent world knowledge. Empirically, SCOUT enables a Qwen2.5-3B-Instruct model to achieve an average score of 0.86, significantly outperforming proprietary models, including Gemini-2.5-Pro (0.60), while saving about 60% GPU hours consumption.
BibTeX
@inproceedings{wang2026language,
  author = {Haoyu Wang and Guozheng Ma and Shugang Cui and Yilun Kong and Haotian Luo and Li Shen and Mengya Gao and Yichao Wu and Xiaogang Wang and Dacheng Tao},
  title = {{Language-based Trial and Error Falls Behind in the Era of Experience}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=XGbggvR35p},
}
Figure