TL;DR
This paper introduces T$^2$PAM, a novel paradigm for improving Large Language Model (LLM) performance in multi-turn conversations by adapting the model's policy in real-time using user feedback. Their proposed algorithm, ROSA, achieves this adaptation efficiently in a single step, significantly boosting accuracy and self-correction capabilities without requiring costly retraining.
What Problem Does This Paper Address?
Large Language Models (LLMs) are increasingly used in multi-turn conversations to complete complex tasks. However, LLMs are typically trained on static, single-turn data, which limits their ability to adapt to evolving user preferences or correct mistakes during extended interactions. This mismatch between training data and real-world usage leads to performance degradation and a gap between benchmark results and practical utility. Existing methods like prompt engineering and retrieval-augmented generation have limitations in efficiency or effectiveness for this specific problem.
Methods
The paper proposes Test-Time Policy Adaptation for Multi-Turn Interactions (T$^2$PAM). This paradigm shifts policy adaptation from offline training to the inference stage. The core idea is to use user feedback (as a reward signal) during a conversation to estimate a latent optimal policy and then update a small subset of the LLM's parameters to steer it towards this policy.
To operationalize T$^2$PAM, they introduce Optimum-Referenced One-Step Adaptation (ROSA). ROSA is a lightweight algorithm that:
- Estimates an optimal policy: It uses a theoretical closed-form solution derived from a Reinforcement Learning from Human Feedback (RLHF) objective, incorporating KL regularization to maintain stability.
- Performs a single-step update: It analytically computes a parameter update (Δθ) to move the model's current policy towards this estimated optimal policy. This is achieved via linearized optimization using the Conjugate Gradient method, avoiding computationally expensive iterative updates.
This approach allows for efficient, in-conversation self-correction with minimal computational overhead.
Datasets
The authors evaluate ROSA on a diverse set of benchmarks across four problem-solving domains:
- Mathematical Reasoning: MATH, AIME25, MATH-500
- General Reasoning: GPQA-diamond, MMLU-Redux, SuperGPQA
- Code Generation: HumanEval
- Multilingual Reasoning: MCLM
Key Insights and Ideas
- Paradigm Mismatch: The core issue is that LLMs trained on static, single-turn data struggle in dynamic, multi-turn interactions where user intent evolves.
- Test-Time Adaptation is Key: Adapting the LLM's policy during the conversation, rather than relying solely on pre-training, is crucial for effective multi-turn performance.
- Efficiency Matters: Real-time adaptation needs to be computationally lightweight to avoid noticeable latency. ROSA's single-step update is a significant advantage here.
- Theoretical Guarantees: The paper provides theoretical analysis showing that ROSA's adaptation process provably reduces error and converges towards the user's preferences.
- Robustness to Feedback Granularity: ROSA shows significant improvements even with sparser, rule-based feedback, highlighting its flexibility.
Key Results
- Significant Accuracy Gains: ROSA consistently outperforms baseline methods across various LLMs and datasets, achieving substantial accuracy improvements. For example, on the MATH dataset, ROSA achieved a +24.00% accuracy improvement over the RL baseline for the Qwen3-0.6B model.
- Enhanced Self-Correction: ROSA dramatically improves the "Correction Uplift" metric, meaning it's much better at recovering from initial errors. For instance, on MATH-500, ROSA boosted the Correction Uplift score by +31.31% for the Qwen3-0.6B model.
- Efficiency: Despite performing updates during inference, ROSA's time-to-accuracy curves show it quickly overtakes baselines, demonstrating its efficiency in achieving high accuracy over time. It also incurs negligible GPU memory overhead.
- Comparison to Training-Based Methods: ROSA achieves performance comparable to or even superior to traditional multi-turn training methods (SFT and RL) but without the need for expensive data collection and training.
Limitations and Future Work
- Approximation Error: While ROSA is efficient, its parameter update relies on a linearized approximation, which introduces a small error. The theoretical analysis accounts for this, but it remains a factor.
- Hyperparameter Sensitivity (Limited): While the study shows ROSA is robust to the regularization coefficient β across a wide range, optimal tuning might still be beneficial for specific scenarios.
- Future Work: The paper suggests exploring more sophisticated adaptation strategies and reward modeling techniques to further enhance performance and efficiency. The T$^2$PAM paradigm itself opens avenues for developing new test-time adaptation methods.
Summary updated