MP2: A Momentum Contrast Approach for Recommendation with Pointwise and Pairwise Learning

Menghan Wang, Yuchen Guo, Zhenqi Zhao, Guangzheng Hu, Yuming Shen, Mingming Gong, Philip Torr

Deep learning has revolutionized how we build recommendation systems, but it still struggles with a fundamental flaw in its data: binary labels are too simple. When we train a model on "clicks" or "purchases," we treat every interaction as a 1 and every non-interaction as a 0. This ignores the nuanced reality of human preference.

In their paper, "MP2: A Momentum Contrast Approach for Recommendation with Pointwise and Pairwise Learning," researchers propose a novel framework to handle this "annotation bias" by blending absolute (pointwise) and relative (pairwise) learning with a clever momentum-based stabilization technique.

TL;DR

MP2 addresses "annotation bias"—where binary labels like clicks fail to capture nuanced user preferences—by combining pointwise and pairwise learning in a novel three-tower architecture. It uses momentum updates and discrepancy-based loss weighting to create stable item representations that are robust to noisy, conflicting feedback.

What Problem Does This Paper Address?

Most modern recommenders rely on implicit feedback (e.g., did the user click?). These are "pointwise" labels: 1 for a click, 0 for an impression without a click. However, these labels are often "noisy."

Consider a user who likes Item A (90% preference), Item B (70%), and Item C (10%).

  • In one session, they see A and B; they click A but not B. Label: B=0.
  • In another session, they see B and C; they click B but not C. Label: B=1.
Figure 1: An example of one user’s two interaction records and the corresponding pointwise and pairwise labels. The pointwise labels of item 𝐵have an annotation bias problem.

This is annotation bias. The model receives contradictory signals for Item B, leading to high fluctuations in learned representations and poor performance. While pairwise labels (e.g., "User prefers A over B") are more robust because they capture relative order, they are harder to scale and implement effectively in deep learning backbones.

Methods

The authors introduce MP2, which stands for Momentum Pointwise and Pairwise learning. The core of the approach is a three-tower architecture:

  1. User Network ($\phi_u$): Generates user representations.
  2. Item Vanilla Network ($\phi_v$): A standard network updated via backpropagation.
  3. Item Momentum Network ($\phi_m$): A "slow-moving" version of the vanilla network. Its parameters are updated as a moving average of the vanilla network's parameters ($\theta_m = \alpha\theta_m + (1-\alpha)\theta_v$).
Figure 2: Graphical architecture of MP2.

Key Innovations:

  • Momentum-Stabilized Pointwise Learning: Instead of using the "jittery" vanilla representations for the noisy pointwise task, MP2 uses the smoother momentum representations. This prevents the model from overreacting to conflicting 0/1 labels.
  • Discrepancy Weighting: The model calculates the "discrepancy" (difference) between the vanilla and momentum representations of an item. If they differ significantly, it signals that the model is uncertain about that item. MP2 then automatically down-weights that sample's contribution to the pointwise loss, effectively telling the model to "trust" noisy labels less.
  • Joint Loss: The final objective function combines a weighted pointwise cross-entropy loss with a pairwise ranking loss, ensuring the model learns both absolute interest and relative preference.

Datasets

The authors evaluated MP2 on four widely used benchmark datasets:

  • MovieLens-100k & MovieLens-1m: Classic movie recommendation datasets.
  • Amazon Beauty & Amazon Office Products: E-commerce datasets containing product reviews and metadata.

Pointwise labels were created by binarizing ratings (e.g., rating $\geq$ 3 is a 1), and pairwise labels were generated by comparing relative ratings of item pairs for the same user.

Key Insights and Ideas

  • Momentum as a Denoiser: While momentum is popular in contrastive learning (like MoCo) for maintaining a large dictionary of samples, here it serves a different purpose: acting as a temporal filter to smooth out the "optimization noise" caused by conflicting labels.
  • Uncertainty via Divergence: The idea that the gap between a "fast" model and a "slow" model represents uncertainty is a powerful heuristic. It allows for automated sample weighting without needing complex Bayesian machinery.
  • Complementary Strengths: Pointwise learning is good for capturing the absolute magnitude of interest, while pairwise learning is superior for ranking. MP2 proves that a structural combination of both—rather than just a mixed loss function—is key to state-of-the-art results.

Key Results

MP2 consistently outperformed competitive baselines like NeuMF (pointwise) and BPR (pairwise) across all datasets.

Table 1: Performance comparison showing MP2 outperforming baselines across four datasets.
  • Significant Gains: On MovieLens-1m, MP2 achieved an NDCG@20 of 0.5314, significantly higher than NeuMF (0.3002) and BPR (0.5241).
  • Sensitivity to Momentum: The experiments showed that a high momentum coefficient (around $\alpha = 0.999$) is optimal. This confirms that a very "slow" and stable item representation is necessary to combat annotation bias.
Figure 3: Performance of different momentum coefficient 𝛼.

Limitations and Future Work

While MP2 shows strong results, the authors note a few areas for growth:

  • Listwise Learning: The current framework only considers pairs. Moving to listwise learning (ranking a whole set of items at once) could further improve results.
  • Computational Overhead: Adding a third tower increases the parameter count and memory footprint, though the momentum update itself is computationally cheap compared to backpropagation.
  • Cold Start: The paper doesn't explicitly discuss how the momentum approach handles extremely sparse "cold-start" items where representations might not have enough updates to stabilize.

Summary updated

Abstract

Binary pointwise labels (aka implicit feedback) are heavily leveraged by deep learning based recommendation algorithms nowadays. In this paper we discuss the limited expressiveness of these labels may fail to accommodate varying degrees of user preference, and thus lead to conflicts during model training, which we call annotation bias. To solve this issue, we find the soft-labeling property of pairwise labels could be utilized to alleviate the bias of pointwise labels. To this end, we propose a momentum contrast framework (MP2) that combines pointwise and pairwise learning for recommendation. MP2 has a three-tower network structure: one user network and two item networks. The two item networks are used for computing pointwise and pairwise loss respectively. To alleviate the influence of the annotation bias, we perform a momentum update to ensure a consistent item representation. Extensive experiments on real-world datasets demonstrate the superiority of our method against state-of-the-art recommendation algorithms.
BibTeX
@inproceedings{wang2022mp,
  author = {Menghan Wang and Yuchen Guo and Zhenqi Zhao and Guangzheng Hu and Yuming Shen and Mingming Gong and Philip Torr},
  title = {{MP2: A Momentum Contrast Approach for Recommendation with Pointwise and Pairwise Learning}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2022},
}
Figure