Disentangling User Interest and Conformity for Recommendation with Causal Embedding

Yu Zheng, Chen Gao, Xiang Li, Xiangnan He, Yong Li, Depeng Jin

Abstract

Modern recommender systems often struggle to distinguish between a user's genuine interest in an item and their tendency to follow trends, known as conformity. This entanglement leads to biased models that perform poorly when popularity distributions shift. We present DICE (Disentangling Interest and Conformity with Causal Embedding), a framework that leverages causal inference to separate these factors into distinct embeddings. By utilizing the colliding effect to partition training data and employing multi-task curriculum learning, DICE achieves superior robustness and interpretability. Experimental results on large-scale datasets show significant improvements over state-of-the-art debiasing methods.

Introduction

Recommendation models typically rely on observational data such as clicks and purchases. However, these interactions are driven by two distinct latent causes: Interest (the user's inherent preference) and Conformity (the influence of item popularity). Standard models represent these causes with a single unified embedding, which creates two primary challenges. First, models become brittle; if a recommendation is based on transient popularity, it fails when that popularity fades. Second, the lack of interpretability makes it difficult to discern whether a recommendation is a personalized discovery or a generic popular choice.

Illustration of the distribution shift between training and test data in recommendation systems.

Existing debiasing techniques often treat popularity as a static global bias. In contrast, DICE recognizes that conformity is user-specific—different users react differently to popular items.

Methodology: The DICE Framework

DICE addresses the entanglement problem through structural disentanglement and causal data mining.

Structural Disentanglement

Instead of a single vector, DICE assigns two separate embeddings for each user and item: one for interest and one for conformity. The final matching score is calculated as the sum of an interest-based score and a conformity-based score.

The DICE framework architecture for causal embedding disentanglement.

Causal Data Mining via the Colliding Effect

A key challenge is the lack of explicit labels for "interest" vs. "conformity." DICE overcomes this by applying the colliding effect from causal inference to partition the training data. For instance, if a user clicks an unpopular item while ignoring a popular one, it provides a strong signal that their interest in the unpopular item outweighs their interest in the popular one. By identifying such inequalities, the model can be trained on samples where one cause is likely dominant.

Multi-task Curriculum Learning

The model is optimized using four simultaneous tasks:

  1. Interest Modeling: Learning preferences from interest-dominant samples.
  2. Conformity Modeling: Capturing popularity-driven behavior.
  3. Click Estimation: Predicting the overall interaction probability.
  4. Discrepancy Loss: Ensuring the interest and conformity embeddings capture distinct information.

A curriculum strategy is employed, starting with "easy" samples (large popularity gaps) and gradually introducing "harder" ones to stabilize training.

Experimental Evaluation

DICE was evaluated on the Movielens-10M and Netflix Prize datasets. To assess robustness against popularity bias, the authors created an "intervened" test set where items were sampled with equal probability, simulating a popularity-neutral environment.

Key Results

  • Performance: DICE outperformed state-of-the-art baselines like IPS and CausE by over 15% in Recall and NDCG on Movielens-10M.
  • Robustness: While standard models saw a sharp decline in accuracy on the intervened test set, DICE maintained stable performance, indicating it successfully learned underlying user interests.
  • Interpretability: t-SNE visualizations confirmed that conformity embeddings cluster by item popularity, while interest embeddings remain mixed, demonstrating successful disentanglement.

Conclusion and Future Work

DICE provides a robust framework for disentangling latent causes in recommendation systems. While it assumes an additive relationship between interest and conformity, future work could explore more complex non-linear interactions and finer-grained interest factors such as brand or price. Despite the increased computational cost of multi-task training, the gains in robustness and interpretability make DICE a significant advancement for real-world recommender systems.

Summary updated

Abstract

Recommendation models are usually trained on observational interaction data. However, observational interaction data could result from users’ conformity towards popular items, which entangles users’ real interest. Existing methods tracks this problem as eliminating popularity bias, e.g., by re-weighting training samples or leveraging a small fraction of unbiased data. However, the variety of user conformity is ignored by these approaches, and different causes of an interaction are bundled together as unified representations, hence robustness and interpretability are not guaranteed when underlying causes are changing. In this paper, we present DICE, a general framework that learns representations where interest and conformity are structurally disentangled, and various backbone recommendation models could be smoothly integrated. We assign users and items with separate embeddings for interest and conformity, and make each embedding capture only one cause by training with cause-specific data which is obtained according to the colliding effect of causal inference. Our proposed methodology outperforms state-of-the-art baselines with remarkable improvements on two real-world datasets on top of various backbone models. We further demonstrate that the learned embeddings successfully capture the desired causes, and show that DICE guarantees the robustness and interpretability of recommendation.
BibTeX
@inproceedings{zheng2021disentangling,
  author = {Yu Zheng and Chen Gao and Xiang Li and Xiangnan He and Yong Li and Depeng Jin},
  title = {{Disentangling User Interest and Conformity for Recommendation with Causal Embedding}},
  booktitle = {The Web Conference},
  year = {2021},
  doi = {10.1145/3442381.3449788},
}
Figure