Self-Improving LLM Agents at Test-Time

TL;DR

This paper introduces Test-Time Self-Improvement (TT-SI), a framework that allows LLM agents to adapt to challenging tasks on-the-fly during inference. By identifying uncertain queries, generating similar synthetic training data, and performing rapid, temporary fine-tuning, agents achieve significant performance gains—averaging a +5.48% accuracy boost—while using 68x less data than traditional supervised fine-tuning.

What Problem Does This Paper Address?

Traditional Large Language Model (LLM) training follows a "train-then-test" paradigm: models are fine-tuned on massive, static datasets and then deployed. This approach has three major flaws:

1. Inefficiency: Training on huge datasets is expensive and often redundant, as models spend compute learning things they already know.
2. Distribution Shift: Models often struggle when test-time queries differ from their training data.
3. Catastrophic Forgetting: Updating a model on new tasks often degrades its performance on old ones.

The authors argue that agents should behave more like humans: when we encounter a difficult homework problem, we don't re-read the whole textbook; we find similar practice problems to sharpen our understanding of that specific concept.

Methods

The authors propose a three-stage algorithm that operates entirely at test-time:

1. Self-Awareness (Uncertainty Estimation): The model evaluates a query $x_i$. Using a "Relative Softmax Scoring" (RSS) mechanism, it calculates the confidence difference between its top two predicted actions. If the gap is below a threshold $\tau$, the model flags the query as "uncertain."
2. Self-Augmentation (Data Synthesis): For uncertain queries, the model uses the input as a seed to generate $K$ similar synthetic examples (input-output pairs). The paper shows that even $K=1$ (a single synthetic sample) is remarkably effective.
3. Self-Improvement (Test-Time Fine-tuning): The model performs a lightweight, temporary update using LoRA (Low-Rank Adaptation) on these synthetic samples. Crucially, these weights are reset after the final answer is produced, preventing permanent model drift or forgetting.

Expand

The researchers also explored Test-Time Distillation (TT-D), where a stronger "teacher" model (like GPT-4o-mini) generates the synthetic training samples instead of the agent itself.

Datasets

The framework was evaluated on four diverse agentic benchmarks:

• NexusRaven: Focuses on complex function calling (nested and parallel calls).
• SealTool: A large-scale tool-learning dataset designed to minimize data leakage.
• API-Bank: Evaluates multi-turn dialogues and conversational tool use.
• ToolAlpaca: A benchmark for generalized tool-use capabilities across 50 categories.

Key Insights and Ideas

• Sharpening, Not Learning: The authors posit that self-improvement works by "sharpening" the model's existing latent knowledge. It doesn't create knowledge from nothing; it re-weights the model's internal probability distribution to favor correct reasoning paths it already "knew" but couldn't confidently access.
• The Power of One: One of the most surprising findings is that training on just one well-chosen synthetic sample is enough to resolve uncertainty and boost accuracy.
• Uncertainty Filtering is Essential: Training on every single query (ignoring uncertainty) provides only marginal gains but incurs massive computational costs. Targeting only the "hard" cases is the secret to efficiency.
• Training-Free Alternative: For environments where fine-tuning is too slow, the authors found that simply inserting the synthetic examples into the prompt (Test-Time ICL) also outperforms standard few-shot baselines.

Expand

Key Results

• Broad Gains: TT-SI improved baseline performance across all benchmarks, with absolute accuracy increases ranging from +4.26% (API-Bank) to +6.05% (NexusRaven).
• Efficiency: On the SealTool benchmark, TT-SI achieved 72.43% accuracy using only 190 uncertain cases. This outperformed standard Supervised Fine-Tuning (70.20%), which required the full 13,000-sample training set—a 68x reduction in data.
• Teacher Boost: Using a stronger model for distillation (TT-D) provided an additional ~1% gain over self-improvement, suggesting that higher-quality training signals further enhance the "sharpening" effect.

Expand

Limitations and Future Work

• The Knowledge Ceiling: TT-SI cannot help if the model lacks the underlying knowledge entirely (e.g., a brand-new API released after the model's cutoff). In these cases, retrieval-augmented generation (RAG) would be necessary.
• Threshold Sensitivity: The performance is sensitive to the uncertainty threshold $\tau$. Finding a way to set this threshold autonomously remains an open challenge.
• Latency: While efficient in terms of data, the process adds roughly 7 seconds of latency per uncertain query (for generation and LoRA updates). Future work could focus on optimizing this overhead for real-time applications.
• Co-evolution: The authors suggest a "dual-learning" setup where the agent and the data generator improve each other over time.

@misc{acikgoz2025self,
  author = {Emre Can Acikgoz and Cheng Qian and Heng Ji and Dilek Hakkani-Tür and Gokhan Tur},
  title = {{Self-Improving LLM Agents at Test-Time}},
  year = {2025},
  eprint = {2510.07841},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.07841},
}