ReasonRec: A Reasoning-Augmented Multimodal Agent for Unified Recommendation

Yihua Zhang, Xi Liu, Xihuan Zeng, Mingfu Liang, Jiyan Yang, Rong Jin, Wen-Yen Chen, Yiping Han, Hao Ma, Bo Long, Huayu Li, Buyun Zhang, Liang Luo, Sijia Liu, Tianlong Chen

TL;DR

This paper introduces REASONREC, a novel multimodal recommendation agent that uses a Vision-Language Model (VLM) to reason explicitly about user preferences. By combining chain-of-thought reasoning with an adaptive delegation strategy to classical models, REASONREC achieves significant accuracy improvements and reduces inference latency, making recommendations more interpretable and efficient.

What Problem Does This Paper Address?

Modern multimodal recommender systems, while adept at fusing visual and textual information, often operate as "black boxes." They lack transparency in their decision-making processes and struggle with efficiency, especially in challenging scenarios like cold-start users or sparse data. This opacity hinders interpretability and makes it difficult to understand why a particular recommendation is made. Furthermore, these systems tend to treat all recommendation queries with the same computational intensity, leading to wasted resources. The authors aim to address this by creating a recommender that can reason explicitly, understand its own uncertainty, and intelligently delegate tasks to specialized models.

Methods

REASONREC employs a three-stage pipeline:

  • Observe: A pretrained Vision-Language Model (VLM) encoder processes multimodal inputs (user history, item images, text) to create unified embeddings.
  • Deliberate: This stage is the core innovation. It reformulates recommendation tasks into structured Chain-of-Thought (CoT) reasoning problems using a "Reasoning-Aware Visual Instruction Tuning" (R-VIT) strategy. R-VIT transforms diverse recommendation tasks into unified CoT prompts, allowing the VLM to articulate its intermediate reasoning steps. To handle data sparsity and improve generalization, an "evidence-horizon-aware curriculum learning" strategy is used, gradually increasing the complexity of training data. This curriculum prioritizes users with more interactions early on and then progressively introduces sparser data.
  • Act: The agent dynamically decides whether to handle a query itself or delegate it to lightweight classical recommendation models (tools). This decision is guided by the agent's confidence and the user's "evidence horizon" (a measure of data sparsity). High-confidence, low-sparsity queries might be delegated for efficiency, while uncertain or complex queries are handled by the VLM, potentially with help from specialized tools.

Datasets

The authors evaluated REASONREC on five real-world datasets:

  • Amazon Review Dataset: Used across four categories: Clothing, Shoes & Jewelry, Sports & Outdoors, Beauty, and Toys & Games. This dataset includes user purchase histories, item metadata, textual reviews, and images.
  • Pixel-1M Dataset: A large-scale image-centric dataset with over 1 million users, 100K images, and 20 million user-image interactions. This dataset allows for direct learning from raw image pixels.

Key Insights and Ideas

  • Explicit Reasoning for Recommendations: The paper argues that moving beyond simple feature fusion to explicit, step-by-step reasoning (like CoT) is crucial for building more interpretable and robust recommender systems.
  • Evidence Horizon as a Sparsity Metric: Quantifying user data sparsity with an "evidence horizon" score provides a principled way to adapt the model's complexity and decision-making strategy.
  • Adaptive Delegation for Efficiency: Dynamically delegating queries to specialized, lightweight models based on risk (uncertainty + evidence horizon) is key to balancing accuracy and computational cost. This is a significant departure from treating all queries uniformly.
  • Curriculum Learning for Generalization: Gradually increasing the difficulty of training data (from warm users to cold-start users) is vital for improving the model's ability to generalize to unseen and challenging scenarios.

Key Results

  • Accuracy Improvement: REASONREC achieved over 30% relative improvement in key ranking metrics (HR@5, NDCG@5) compared to state-of-the-art multimodal recommenders across four standard tasks (sequential recommendation, direct recommendation, CTR prediction, explanation generation).
  • Efficiency Gains: The system substantially reduced inference latency by dynamically delegating up to 35% of queries to efficient sub-models without compromising accuracy.
  • Robustness: REASONREC demonstrated strong performance in challenging scenarios like cold-start, time-shift, and multi-domain recommendations, outperforming baselines that struggled with data sparsity and domain shifts.
  • Ablation Studies: All proposed components (R-VIT, evidence-horizon curriculum, uncertainty-guided delegation) were shown to be crucial, with curriculum learning having the largest impact.

Limitations and Future Work

  • Reliance on VLMs and Templates: The system heavily relies on pretrained VLMs and carefully crafted instruction templates. Adapting to entirely new tasks or domains might require significant template engineering.
  • Preselected Classical Models: The tool delegation mechanism uses a fixed set of classical models. Generalizing this to unseen recommendation scenarios or emerging tasks could be a challenge.
  • Inference Overhead: While improved, the multi-stage reasoning and dynamic routing still introduce some non-trivial overhead compared to simpler models.
  • Ethical Considerations: The authors acknowledge that the model inherits general concerns related to large AI systems, such as potential biases in training data and fairness. Future work should focus on fairness-aware adaptations and auditing techniques. They also emphasize the need for responsible deployment and transparency as recommendation systems become more influential.

Summary updated

Abstract

Recent advances in multimodal recommenders excel at feature fusion but remain opaque and inefficient decision-makers, lacking explicit reasoning and self-awareness of uncertainty. To address this, we introduce ReasonRec, a reasoning-augmented multimodal agent structured around a three-stage explicit reasoning pipeline: Observe, via a pretrained Vision-Language Model (VLM) encoder; Deliberate, by formulating recommendation as chain-of-thought (CoT) reasoning tasks and explicitly quantifying prediction uncertainty through an evidence-horizon-aware curriculum; and Act, through dynamic delegation of uncertain or challenging queries to lightweight classical recommendation models. Specifically, we propose a reasoning-aware visual instruction tuning strategy that systematically transforms diverse recommendation tasks into unified CoT prompts, enabling the VLM to explicitly articulate intermediate decision steps. Additionally, our evidence-horizon curriculum progressively enhances the reasoning complexity to better handle cold-start and long-tail user scenarios, significantly boosting model generalization. Furthermore, the uncertainty-guided delegation mechanism empowers the agent to assess its own confidence, strategically allocating computational resources to optimize both recommendation accuracy and inference efficiency. Comprehensive experiments on four standard recommendation tasks (sequential recommendation, direct recommendation, CTR prediction, and explanation generation) across five real-world datasets demonstrate that ReasonRec achieves over 30% relative improvement in key ranking metrics (e.g., HR@5, NDCG@5) compared to state-of-the-art multimodal recommenders. Crucially, ReasonRec substantially reduces inference latency by dynamically delegating up to 35% of queries to efficient sub-models without compromising accuracy. Extensive ablation studies further confirm that each proposed reasoning and planning mechanism individually contributes substantially to ReasonRec's overall effectiveness. Collectively, our results illustrate a clear pathway towards interpretable, adaptive, and efficient multimodal recommendation through explicit reasoning and agentic design.
BibTeX
@inproceedings{zhang2026reasonrec,
  author = {Yihua Zhang and Xi Liu and Xihuan Zeng and Mingfu Liang and Jiyan Yang and Rong Jin and Wen-Yen Chen and Yiping Han and Hao Ma and Bo Long and Huayu Li and Buyun Zhang and Liang Luo and Sijia Liu and Tianlong Chen},
  title = {{ReasonRec: A Reasoning-Augmented Multimodal Agent for Unified Recommendation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.391},
}
Figure