ReflectRec: Cycle-Consistency Preference Verification for LLM-based Recommendation

Dae Kyoo Kim, Heejung Cho, Sungmoo Hong, Eunil ParkarXiv 2026 · DOI · arXiv

TL;DR

ReflectRec introduces a "cycle-consistency" verification step to ensure that LLM-generated user preferences are actually faithful to a user's history. By combining this self-verification with a "Forward-Look" retrieval mechanism that injects collaborative signals from similar users, the system significantly outperforms existing Chain-of-Thought (CoT) recommenders across multiple benchmarks.

What Problem Does This Paper Address?

Large Language Models (LLMs) are increasingly used for recommendation because they can reason about why a user might like an item, often generating "preference notes" as a chain-of-thought prefix to ranking. However, two major issues persist:

  1. Lack of Verification: LLMs are prone to hallucinations. They might generate a beautifully written preference note that sounds plausible but has nothing to do with the user's actual viewing history. Current systems trust these notes blindly.
  2. Missing Collaborative Cues: Traditional recommendation systems excel at finding "users like you." LLM-based systems often focus solely on a single user's history, missing out on the powerful local collaborative signals found in what behaviorally similar users consumed next.

Methods

The authors propose a four-stage pipeline called ReflectRec, centered around a novel "round-trip" verification process.

  1. Semantic Preference Elicitation (Stage 1): The LLM analyzes a user's history to describe their taste along five semantic dimensions (e.g., genre, mood, narrative style) without naming specific titles.
  2. Cycle-Consistency Verifier: This is the "aha!" moment of the paper. To check if the preference note ($P_u$) is accurate, the system feeds it back to the LLM along with a mix of 50 candidates (the user's true history items + random negatives). The LLM must pick which items the user actually watched based only on the note. The recall score (ValScore) serves as a measure of "recoverability."
  3. Conditional PV-guided Regeneration: If the ValScore is low (below a threshold $\tau=0.4$), the system triggers a regeneration of the preference note. It attempts this up to two times and keeps the version with the highest ValScore.
  4. Forward-Look Retrieval: To bring back collaborative filtering, the system maintains a "fragment pool" of other users' histories. It retrieves fragments from users with similar past histories and injects their future watches into the prompt as "collaborative hints."
Figure 1: REFLECTREC overview. Stage-1 elicits a structured user preference Pu; the cycle-consistency verifier yields a score of preference-to-history recoverability, which gates regeneration, selects the regeneration attempt with the highest recoverability, and is used to analyze downstream module effects. Conditional regeneration regenerates low-recoverability preferences, and Forward-Look retrieval injects the future watches of behaviorally similar users into the ranker prompt.

Datasets

The researchers evaluated ReflectRec on three primary datasets:

  • MovieLens-100K & MovieLens-1M: Standard benchmarks for movie recommendation.
  • Amazon Electronics (ELEC): Used for cross-domain validation to see if the verification logic holds up in a sparser, more technical product domain.

Key Insights and Ideas

  • ValScore as a Multi-tool: The cycle-consistency score isn't just a metric; it's an operational signal used for gating (deciding when to regenerate), selection (picking the best attempt), and diagnosis (understanding where the model fails).
  • Grounded Verification vs. LLM-as-a-Judge: Instead of asking an LLM "Is this a good summary?" (which is prone to sycophancy), ReflectRec uses a task-grounded "round-trip" check. If the summary can't help recover the original data, it's a bad summary.
  • Complementary Gains: The authors found that different modules help different users. Regeneration is a lifesaver for the "LOW" bucket (users with vague initial preferences), while Forward-Look retrieval provides the biggest boost for users who already have high-quality preference notes.
  • Alignment of Retrieval: Unlike standard RAG which might retrieve items similar to the preference, "Forward-Look" retrieves items based on history similarity but provides the future items as hints. This aligns the retrieval target directly with the next-item prediction task.
Figure 2: Training users’ histories are split into history h and future f fragments to build a fragment pool (Step 1). A new user’s last 10 interactions are embedded via BGE-large and matched to the top-K similar users by cosine similarity (Step 2). Only the future fragments of matched users are extracted and injected as next-watch hints into the LLM ranker prompt alongside the user’s history and preferences (Step 3).

Key Results

ReflectRec demonstrates substantial improvements over the state-of-the-art CoT4Rec-ICL:

  • Hit@10 Gains: Improved by +16.25 percentage points (pp) on MovieLens-100K and +4.46 pp on MovieLens-1M.
  • Backbone Robustness: The gains were consistent across different LLM backbones, including Qwen3-30B, Qwen2.5-72B, and Llama-3.3-70B.
  • Ablation: The "Stage-1" semantic preference alone provided the largest single jump (+12.34 pp on ML-100K), but the Verifier and Forward-Look modules provided critical additive gains, especially for difficult samples.
  • Cross-Domain: On Amazon Electronics, the verifier-guided regeneration pattern held true, proving the method isn't just "overfit" to movie descriptions.

Limitations and Future Work

  • Threshold Tuning: The optimal ValScore threshold ($\tau=0.4$) was calibrated for Qwen3-30B. The authors note that this might need recalibration for different models or candidate pool sizes.
  • Temporal Sensitivity: In a strict "timestamp-safe" evaluation (where you can't look at any data from the future relative to the query), the gains from Forward-Look on the larger ML-1M dataset were not statistically significant, suggesting a need for more robust temporal modeling.
  • Manual Axis Design: The semantic axes (genre, mood, etc.) were hand-designed for each domain. Future work could involve an automated process to select the most discriminative axes for any given dataset (e.g., books or research papers).

Summary updated

Abstract

CoT-style LLM recommenders generate a natural-language preference note as a chain-of-thought prefix to ranking, but the note is trusted without verification, and the ranker never consults behaviorally similar users. We propose ReflectRec, organized around a preference-to-history round-trip verifier whose recall score serves as an operational signal of preference-to-history recoverability, enabling conditional regeneration for low-recoverability preferences and selection among regeneration attempts. Around this verifier, we wire a semantic Stage-1 prompt, conditional regeneration, and Forward-Look retrieval that injects future watches of behaviorally similar users as a collaborative hint. On MovieLens-100K and MovieLens-1M, the full pipeline outperforms CoT4Rec-ICL by +16.25 / +4.46 pp Hit@10, with positive gains across all ValScore buckets and three LLM backbones (Qwen3-30B, Qwen2.5-72B, and Llama-3.3-70B). We also observe a similar verifier-guided regeneration pattern on Amazon Electronics.
BibTeX
@misc{kim2026reflectrec,
  author = {Dae Kyoo Kim and Heejung Cho and Sungmoo Hong and Eunil Park},
  title = {{ReflectRec: Cycle-Consistency Preference Verification for LLM-based Recommendation}},
  year = {2026},
  eprint = {2611.23969},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23969},
}
Figure