Teaching the Harness: Student-Aware Skill Transfer for LLM Agents

Chang Liu, Bo WuarXiv 2026 · arXiv

TL;DR

Researchers have developed Student-Aware Teaching (SAT), a framework where a strong "teacher" LLM improves a "student" LLM agent by explicitly modeling the student's specific weaknesses. By using diagnostic probes to find where a student hits its limits—such as running out of steps in a tool-use loop—the teacher writes tailored "skills" that help the student navigate its environment more effectively, even after the teacher is gone.

What Problem Does This Paper Address?

Most LLM agents today don't just exist as a raw model; they operate within a harness—a software layer that provides tools, manages memory, and sets an "action budget" (e.g., a limit of 10 steps to solve a task).

The challenge is that a strategy that works for a frontier model like GPT-4 might cause a smaller model to fail. For instance, a "verify everything" strategy might be great for a reliable model but could cause a slower, more repetitive model to exhaust its action budget before finishing the actual task. Traditional "machine teaching" often focuses on generic prompt optimization or weight distillation, ignoring the fact that many agent failures are actually failures of execution within these specific harness constraints.

The authors ask: Can a teacher produce better results if it actually understands the specific student it is teaching?

Methods

The authors introduce Student-Aware Teaching (SAT), which implements a "Theory of Mind" approach to AI instruction. Instead of just looking at task failures, the teacher (Claude-3.5-Sonnet) treats the student (e.g., DeepSeek-V4-Flash) as a learner with unique latent limitations.

The SAT process follows a five-step loop:

  1. Observe: The teacher watches the student attempt tasks and records traces.
  2. Infer: The teacher builds a Learner Model consisting of an empirical profile (success stats) and analytical hypotheses (e.g., "I suspect the student fails when the inventory list exceeds 5 items").
  3. Test: The teacher uses Diagnostic Probes—active experiments that vary one property of a task (like file count or list size)—to find the student's exact breaking point.
  4. Teach: The teacher writes natural-language "skills" (reusable procedures) specifically designed to bypass the diagnosed failures.
  5. Verify: A skill only enters the student's permanent library if it passes an "admission test" on development tasks.
Figure 1: Student-Aware Teaching. The learner is a student model operating through its existing harness. During preparation the teacher observes the student, infers a learner model, tests it with diagnostic probes, writes skills for the diagnosed limits, and admits only skills that help the student on matched tasks. At deployment the teacher is absent and a selector delivers the saved skills. The agnostic baseline shares the loop but omits explicit learner modeling and diagnostic probing.

Datasets

The researchers evaluated SAT using five diverse task suites designed to trigger different types of agent failures:

  • Terminal: Shell tasks like recursive renaming or file pipelines (tests environment coverage).
  • Inventory API: Restocking items via tools (tests "action economy" and budget management).
  • Arithmetic: Complex word problems (tests logic and semantics).
  • Python Code: Function implementation and bug repair.
  • Dependent Workflows: Multi-stage file processing (tests state tracking).

They also validated the approach on subsets of the MATH and GSM8K benchmarks and tested portability across production harnesses like Claude Code and OpenCode.

Key Insights and Ideas

  • Theory of Mind Matters: Teaching is significantly more effective when the teacher can predict how a student will fail. SAT teachers were much better at predicting student success (measured by lower Brier scores) than "agnostic" teachers who didn't model the student.
  • The "Breaking Point" is the Lesson: By using probes to find exactly where a student fails (e.g., "this model can handle 8 items but fails at 12"), the teacher can write highly specific advice, like suggesting a "bulk query" instead of checking items one by one to save the action budget.
  • Skills are Student-Specific: A library of skills built for DeepSeek doesn't work as well for Gemini, and vice versa. When models "swapped" libraries, their performance dropped, proving that the lessons were truly tailored to the individual model's quirks.
  • The "Goldilocks" Zone for Students: Mid-sized models benefit the most. Very small models (like Qwen-1.7B) often lack the instruction-following capability to use the skills, while frontier models are already near the performance ceiling.

Key Results

In the primary experiments with Claude-Sonnet-5 teaching DeepSeek-V4-Flash:

  • SAT reached 86.46% success, outperforming an "agnostic" teacher (81.04%) and the untaught student (75.00%).
  • Persistence: The gains remained even after the teacher was removed. Saved libraries provided average gains of up to 11.1 points.
  • Efficiency: SAT students often produced significantly fewer output tokens than untaught students because they were following more efficient, teacher-prescribed procedures.
Figure 2: Eight-round extended teaching with Claude-Sonnet-5 and DeepSeek-V4-Flash. Left: test acceptance on 150 held-out tasks after each round. Right: the teacher’s Brier score for predicting student success on a fixed ten-task panel (lower is better).

Limitations and Future Work

  • Student Capability: The student must be "teachable"—it needs a baseline ability to follow natural language instructions. The authors found that the smallest models they tested could not effectively utilize the skill library.
  • Ceiling Effects: For very strong students already performing at 90%+, the added value of SAT is marginal.
  • Teacher Cost: Modeling the student and running diagnostic probes adds to the initial "preparation" cost, though the authors argue this is a one-time investment that pays off during deployment.
  • Future Directions: The authors suggest exploring "self-teaching" (where a model tries to diagnose its own harness failures) and testing SAT in more complex, real-world production environments.

Summary updated

Abstract

Effective teaching requires reasoning about the learner as well as the task, a capacity that Theory of Mind describes as inferring another agent's latent state from its behavior. We ask whether a stronger LLM teaches a fixed student better when it applies this principle. The student is an LLM agent acting through a harness that fixes its tools, context, and action budget, so a procedure that helps one model can exhaust the budget of another. We introduce Student-Aware Teaching (SAT). The teacher builds a learner model that pairs an empirical profile with hypotheses about the student's limitations, tests these hypotheses with diagnostic probes, and writes reusable skills for the diagnosed failures. A skill enters the library only after it raises success on matched development tasks. With Claude-Sonnet-5 teaching DeepSeek-V4-Flash, SAT reaches 86.46% task success across four primary runs, leading both a profile-free teacher with the same framework (81.04%) and the untaught student (75.00%) in each run. Removing the analytical learner model lowers both prediction and success, and under misleading learner profiles none of the 48 candidate skills passes admission. When DeepSeek-V4-Flash and Gemini-3.8-Flash exchange libraries, each performs better with its own. Saved libraries keep average gains of up to 11.1 points after the teacher leaves. The benefit depends on a student that can follow instructions and a deployment route that delivers them.
BibTeX
@misc{liu2026teaching,
  author = {Chang Liu and Bo Wu},
  title = {{Teaching the Harness: Student-Aware Skill Transfer for LLM Agents}},
  year = {2026},
  eprint = {2611.13144},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.13144},
}
Figure