Test-Time Policy Adaptation for Enhanced Multi-Turn Interactions with LLMs

Chenxing Wei, Hong Wang, Ying He, Fei Yu, Yao Shu

TL;DR

While Large Language Models (LLMs) excel at single-turn tasks, their performance frequently degrades across extended conversations because they treat user feedback as passive context rather than an active signal to correct their internal policy. This paper introduces ROSA (Optimum-Referenced One-Step Adaptation), a lightweight test-time adaptation algorithm that analytically calculates an optimal target policy from real-time user feedback and updates a tiny fraction of model parameters in a single, efficient step. ROSA dramatically boosts an LLM's ability to self-correct in-conversation with negligible memory overhead, matching or exceeding the performance of expensive offline training methods.

What Problem Does This Paper Address?

When humans collaborate, we use multi-turn dialogue to progressively clarify intent and correct mistakes. LLMs, however, are predominantly aligned using static, single-turn datasets via Supervised Fine-Tuning (SFT) or Reinforcement Learning from Human Feedback (RLHF).

This training mismatch causes a distinct failure mode in multi-turn interactions: models treat user corrections merely as additional text tokens in their context window rather than as an active directive to adjust their generation strategy. If a model fails on its first attempt, it often gets trapped in a loop of repetitive reasoning errors. Existing test-time workarounds like prompt engineering or Retrieval-Augmented Generation (RAG) either fail to deeply alter the model's behavioral policy or introduce massive inference-time latency and context-length overhead. Meanwhile, full multi-turn offline training is prohibitively expensive.

Methods

The authors formalize a new paradigm called Test-Time Policy Adaptation for Multi-Turn Interactions ($T^2PAM$) and operationalize it via ROSA. Instead of running slow, iterative gradient descent loops during a live conversation, ROSA updates the model using a highly efficient, closed-form approach:

  1. Analytical Target Formulation: ROSA defines a turn-wise RLHF objective that maximizes user reward while penalizing divergence from the previous turn's policy (using KL regularization). The authors derive a closed-form solution showing that the optimal policy is simply an exponentially re-weighted version of the reference policy. When a user provides negative feedback (reward = -1), ROSA instantly scales down the probability of that specific erroneous response.
  2. Linearized One-Step Update: To apply this change without heavy optimization overhead, the policy function is approximated using a first-order Taylor expansion. ROSA then employs the Conjugate Gradient (CG) method to solve the resulting linear system in a single step. This "matrix-free" computation calculates Jacobian-vector products via automatic differentiation, completely avoiding the need to materialize or store a massive Hessian matrix.
  3. Flexible Parameter Targeting: The calculated parameter update ($\Delta \theta$) can be applied in two ways: either by updating the language modeling head using a low-rank adapter (LoRA) or by directly modifying the hidden state activations via transient model hooks.
The T2PAM paradigm treats conversational feedback as an active signal to guide real-time parameter updates.

Datasets

ROSA was rigorously evaluated across four distinct problem-solving domains using several challenging benchmarks:

  • Mathematical Reasoning: MATH, MATH-500, and AIME25.
  • General Reasoning: GPQA-diamond, MMLU-Redux, and SuperGPQA.
  • Code Generation: HumanEval.
  • Multilingual Reasoning: Subsets of the Multilingual Chinese Language Model (MCLM) benchmark, specifically MT-AIME24 and MT-MATH100.

Key Insights and Ideas

  • Provably Productive Failure: The authors provide a rigorous theoretical proof demonstrating Monotonic Error Reduction. They show that every piece of negative feedback under ROSA mathematically guarantees a reduction in the KL divergence between the model's policy and the user's true optimal policy. The model is guaranteed to get closer to the user's intent with every correction.
  • Bypassing the Global Optimization Trap: The core "aha moment" of the paper is realizing that real-time adaptation doesn't require solving an intractable global optimization problem. By isolating the update to a single, sample-wise correction based on the exact response that failed, the global problem collapses into a trivial, targeted adjustment.
  • The Value of Plausible Mistakes: Theoretical analysis reveals that the most impactful learning signals come from correcting plausible mistakes (where the model had high initial confidence but failed) rather than completely nonsensical outputs.

Key Results

  • Dramatic Correction Uplift: ROSA significantly enhances the model's capacity for in-conversation self-correction. On the MATH dataset, Qwen3-0.6B's "Correction Uplift" (the percentage of initially failed problems successfully resolved in later turns) skyrocketed from a baseline of 17.40% to 48.87%.
  • Beating Offline Training with Zero Data Costs: In direct comparisons on the MATH dataset using Qwen3-8B, ROSA achieved a final accuracy of 65.80%, virtually matching full offline Reinforcement Learning (66.20%) and handily defeating Supervised Fine-Tuning (63.80%)—all without requiring any offline dataset collection or pre-training.
  • Highly Efficient Resource Footprint: ROSA introduces negligible memory overhead, requiring at most an additional +1.0 GB of peak GPU memory on the MATH dataset. While calculating updates adds a minor latency bump to the initial turn, ROSA's superior error-correction rate means it achieves higher overall accuracy much faster in terms of total wall-clock time compared to standard prompting baselines.

Limitations and Future Work

  • Reward Model Dependency: ROSA's performance hinges on the quality of its reward signal. The authors utilized both rule-based verifiers and an LLM judge (Qwen3-30B). In real-world deployments, noisy, ambiguous, or incorrect user feedback could steer the parameter updates in the wrong direction.
  • Linearization Errors: The algorithm relies on a first-order Taylor expansion for its speed. While the authors prove that the resulting approximation errors are bounded, these errors can still accumulate over exceptionally long, multi-turn conversations.
  • Future Directions: A natural next step is expanding ROSA to handle richer, non-scalar feedback—such as natural language critiques or multi-modal inputs—and translating them directly into test-time policy updates without relying on an intermediate scalar reward.

Summary updated

Abstract

Large Language Models (LLMs) employ multi-turn interaction as a fundamental paradigm for completing complex tasks. However, their performance often degrades in extended interactions, as they are typically trained on static, single-turn data, which hinders their ability to adapt to real-time user feedback. To address this limitation, we first propose a new paradigm: Test-Time Policy Adaptation for Multi-Turn Interactions (T2PAM), which utilizes user feedback from the ongoing interaction as a reward signal to estimate a latent optimal policy aligned with user preferences, then updates a small subset of parameters to steer the model toward this policy, ultimately enabling efficient in-conversation self-correction. We then introduce Optimum-Referenced One-Step Adaptation (ROSA), a lightweight algorithm that operationalizes T2PAM. ROSA guides the model parameters toward a theoretical optimal policy in a single, efficient update step, avoiding costly iterative gradient-based optimization and minimizing computational overhead. We provide a rigorous theoretical analysis guaranteeing that the policy of ROSA converges to the preference of user as the number of interactions increases. Extensive experiments on challenging benchmark demonstrate that ROSA achieves significant improvements in both task effectiveness and efficiency.
BibTeX
@inproceedings{wei2025test,
  author = {Chenxing Wei and Hong Wang and Ying He and Fei Yu and Yao Shu},
  title = {{Test-Time Policy Adaptation for Enhanced Multi-Turn Interactions with LLMs}},
  booktitle = {MTI-LLM @ NeurIPS},
  year = {2025},
  url = {https://openreview.net/forum?id=6eN5FZJuLI},
}
Figure