TL;DR
CoARS introduces a self-distilled reinforcement learning framework that jointly optimizes both the recommender and user agents within an agentic recommender system. By moving away from purely prompt-based memory and one-sided rewards, it leverages endogenous interaction rewards and token-level credit assignment to bake multi-turn interaction experience directly into the models' parameters.
What Problem Does This Paper Address?
Traditional recommender systems operate in a one-shot fashion, predicting user preferences from static historical data. Emerging Agentic Recommender Systems (ARS) change this by framing recommendation as a multi-turn dialogue between a recommender agent (RecAgent) and a user agent (UserAgent).
However, current ARS architectures face two major bottlenecks:
- In-Context Over-Reliance: Most systems rely on a "Reflexion-style" paradigm, where past interaction trajectories are stored as text and fed back into the prompt context. The models never actually update their underlying weights, meaning they rely on generic reasoning rather than acquiring specialized recommendation skills.
- Sparse, One-Sided Rewards: When researchers do use Reinforcement Learning (RL) to update model parameters, they typically use sparse, outcome-level rewards (e.g., did the user click at the very end?). Furthermore, these setups usually optimize only the recommender agent while keeping the user simulator fixed, failing to capture how both agents continuously influence each other.
To solve this, the authors propose CoARS (Co-evolving Agentic Recommender Systems), an RL framework designed to co-evolve both agents using dense, interaction-driven feedback.
Methods
CoARS abandons external or one-sided reward functions in favor of two complementary learning schemes:
1. Interaction Reward (Task-Level Supervision)
Instead of a single scalar reward, CoARS derives coupled, endogenous rewards for both agents from the exact same interaction trajectory:
- RecAgent Reward: Evaluates recommendation correctness, user acceptance strength, and interaction-stage sensitivity (imposing harsher penalties on mistakes made in later turns of the conversation).
- UserAgent Reward: Encourages the user agent to give realistic feedback by aligning its choices with recommendation correctness, response direction, and peer similarity (derived from a conventional model like SASRec).
2. Self-Distilled Credit Assignment (Token-Level Supervision)
To prevent multi-turn interactions from collapsing into a single final reward, CoARS introduces a self-distillation mechanism for fine-grained token-level updates.
The framework casts the same agent into two modes:
- Student Mode: Sees only the original interaction context.
- Teacher Mode: Sees the original context plus a retrospectively diagnosed "reference trajectory" that highlights what went right or wrong in hindsight.
By comparing the token log-probabilities between the teacher and student modes, the system calculates a token-level diagnostic advantage to guide policy optimization.
Datasets
The authors evaluated CoARS on three sequential recommendation benchmarks:
- LastFM: 2,100 users, 18,745 items, and 92,835 interactions.
- MovieLens: 943 users, 1,682 items, and 100,001 interactions.
- Instruments: A massive dataset containing 903,330 users and 112,136 items. The authors randomly sampled 1,000 data points for the test set to align the evaluation scale with the other datasets.
Key Insights and Ideas
- Parameter-Level Learning Beats Prompt Memory: Moving interaction history out of the prompt context window and directly into the model's weights via RL yields substantial performance gains.
- Co-Evolution is Essential: Optimizing a recommender against a static user simulator creates a bottleneck. Jointly training both agents allows the user agent to become a higher-fidelity simulator while the recommender simultaneously learns to better navigate user feedback.
- Hindsight as a Token-Level Teacher: Generating corrected reference trajectories in hindsight allows the model to perform effective self-distillation, providing dense credit assignment across long conversational turns without needing human annotations.
Key Results
The framework was tested using Qwen3-4B and Qwen3-8B as trainable backbones, comparing against baselines like Reflexion, AFL, iAgent, and RecoWorld.
- Recommendation Performance (Hit@1): CoARS consistently achieved the highest accuracy. On MovieLens using the Qwen3-8B backbone, CoARS achieved a 0.2631 Hit@1, significantly outperforming the RL baseline RecoWorld (0.1724).
- User Simulation Fidelity (F1 Score): CoARS vastly improved the UserAgent's ability to simulate real human feedback. On LastFM, CoARS hit an F1 score of 0.3145, compared to RecoWorld's 0.1513, proving that bidirectional optimization prevents the simulator from stagnating.
Limitations and Future Work
The authors note several open challenges for agentic recommenders:
- Handling Noisy Feedback: The framework needs better robustness to handle uncertain, ambiguous, or misleading user responses.
- Efficiency and Scalability: Running multi-turn interaction loops for parameter-level optimization is computationally heavy. Improving the scalability of this training pipeline is a priority for real-world deployment.
- Safety Alignment: Future work must incorporate explicit safety constraints to ensure the co-evolution loop doesn't accidentally reinforce biased, harmful, or adversarial interaction patterns.
Summary updated