In the standard pipeline for training AI agents, the first step is almost always Behavior Cloning (BC) or Supervised Fine-Tuning (SFT). We show the model a dataset of "good" behavior—human dialogues, robot trajectories, or expert moves in a game—and tell the model: "Do exactly what they did."
This approach is stable and computationally efficient, but it harbors a fundamental flaw: the model becomes a mirror. If the demonstrations are suboptimal, noisy, or merely "average," the model will inherit those exact limitations. This creates a performance ceiling where the student can never become better than the teacher.
Recent research has focused on breaking this plateau, moving from simple mimicry to frameworks that allow agents to "stitch together" optimal strategies from suboptimal data or discover entirely new solutions through environmental feedback.
The Imitation Plateau
The core problem with supervised learning is that it is "mass-covering." As Mihaela van der Schaar and Hao Sun argue in Inverse-RLignment, SFT forces the model to mimic the average behavior of the training set. If a dataset contains a mix of expert and amateur actions, the model learns a mediocre middle ground.
This limitation appears across every domain:
- Robotics: Charles Xu notes in RLDG that generalist policies trained on messy human demonstrations reach a success rate ceiling that more data alone cannot fix.
- Language Models: Geyang Guo and colleagues point out in Beyond Imitation that SFT lacks a "negative signal." The model learns what a good response looks like, but it doesn't understand why it is good or what specific pitfalls to avoid.
- Formal Reasoning: Even in mathematics, Mert Ünsal shows in AlphaIntegrator that mimicking existing symbolic solvers prevents models from discovering more efficient search strategies than the ones already programmed into those solvers.
Stitching: Finding the Best Path Through the Noise
If we cannot rely on a single "perfect" teacher, how do we improve? One powerful intuition is stitching. Imagine a dataset where one person knows how to get from point A to B, and another knows how to get from B to C. Neither knows the full path from A to C, but an agent that can "stitch" these experiences together can outperform both.
Charlie Snell and the team at UC Berkeley addressed this with Implicit Language Q-Learning (ILQL). In Offline RL for Natural Language Generation, they demonstrate that by treating language generation as a reinforcement learning problem, models can maximize utility across a static dataset. Instead of just predicting the next word, the model learns to evaluate which words lead to the highest long-term reward, effectively "stitching" the best parts of various dialogues into a superior policy.
Similarly, Zhishuai Liu and Pan Xu introduced Reinforced RCSL ($R^2CSL$) to provably enable this stitching behavior in supervised learning frameworks, allowing agents to reach the "in-distribution optimal" path even if that path was never seen in its entirety.
From Mimicry to Deliberation
Another way to break the ceiling is to give the agent the ability to think before it acts—moving from instinctive mimicry to active deliberation.
In Pangu-Agent, Filippos Christianos and Matthieu Zimmer formalize "internal thinking" as nested functions. By combining this structured reasoning with reinforcement learning, the agent doesn't just copy a teacher's action; it learns to evaluate the environment and refine its own policy.
This is echoed in the SAND framework by Yu Xia and Tong Yu, which introduces "Self-taught Action Deliberation." By allowing an agent to develop an "inner monologue" to weigh alternatives, it moves past the "look-before-you-leap" errors common in models that only know how to imitate.
Eliciting Latent Capabilities
Perhaps the most surprising finding in this area is that the "student" might already be smarter than the "teacher." In Weak-to-Strong Generalization, Collin Burns and researchers at OpenAI show that a strong model (like GPT-4) can be supervised by a weak model (like GPT-2) and still outperform its supervisor.
The intuition here is that large models have "latent" capabilities—knowledge and reasoning patterns learned during pre-training—that are suppressed when they are forced to mimic a weak supervisor. By using clever regularization, we can "elicit" these superior capabilities rather than just training the model to be a better imitator.
The Path Forward: Self-Optimizing Alignment
The transition from imitation to optimization is becoming the new standard for alignment. Frameworks like GRAO, introduced by Haowen Wang and Yun Yue, bridge the gap by combining the stability of SFT with active self-exploration.
As Minghuan Liu demonstrates with Curriculum Offline Imitation Learning (COIL), we can even build a "curriculum" of increasingly better policies within a single dataset. By iteratively imitating "neighboring" policies that are slightly better than the current one, an agent can climb a ladder of performance, eventually reaching the very best trajectories available and, with the addition of RL, finally stepping beyond them.