Deep learning has revolutionized how we build recommendation systems, but it still struggles with a fundamental flaw in its data: binary labels are too simple. When we train a model on "clicks" or "purchases," we treat every interaction as a 1 and every non-interaction as a 0. This ignores the nuanced reality of human preference.
In their paper, "MP2: A Momentum Contrast Approach for Recommendation with Pointwise and Pairwise Learning," researchers propose a novel framework to handle this "annotation bias" by blending absolute (pointwise) and relative (pairwise) learning with a clever momentum-based stabilization technique.
TL;DR
MP2 addresses "annotation bias"—where binary labels like clicks fail to capture nuanced user preferences—by combining pointwise and pairwise learning in a novel three-tower architecture. It uses momentum updates and discrepancy-based loss weighting to create stable item representations that are robust to noisy, conflicting feedback.
What Problem Does This Paper Address?
Most modern recommenders rely on implicit feedback (e.g., did the user click?). These are "pointwise" labels: 1 for a click, 0 for an impression without a click. However, these labels are often "noisy."
Consider a user who likes Item A (90% preference), Item B (70%), and Item C (10%).
- In one session, they see A and B; they click A but not B. Label: B=0.
- In another session, they see B and C; they click B but not C. Label: B=1.
This is annotation bias. The model receives contradictory signals for Item B, leading to high fluctuations in learned representations and poor performance. While pairwise labels (e.g., "User prefers A over B") are more robust because they capture relative order, they are harder to scale and implement effectively in deep learning backbones.
Methods
The authors introduce MP2, which stands for Momentum Pointwise and Pairwise learning. The core of the approach is a three-tower architecture:
- User Network ($\phi_u$): Generates user representations.
- Item Vanilla Network ($\phi_v$): A standard network updated via backpropagation.
- Item Momentum Network ($\phi_m$): A "slow-moving" version of the vanilla network. Its parameters are updated as a moving average of the vanilla network's parameters ($\theta_m = \alpha\theta_m + (1-\alpha)\theta_v$).
Key Innovations:
- Momentum-Stabilized Pointwise Learning: Instead of using the "jittery" vanilla representations for the noisy pointwise task, MP2 uses the smoother momentum representations. This prevents the model from overreacting to conflicting 0/1 labels.
- Discrepancy Weighting: The model calculates the "discrepancy" (difference) between the vanilla and momentum representations of an item. If they differ significantly, it signals that the model is uncertain about that item. MP2 then automatically down-weights that sample's contribution to the pointwise loss, effectively telling the model to "trust" noisy labels less.
- Joint Loss: The final objective function combines a weighted pointwise cross-entropy loss with a pairwise ranking loss, ensuring the model learns both absolute interest and relative preference.
Datasets
The authors evaluated MP2 on four widely used benchmark datasets:
- MovieLens-100k & MovieLens-1m: Classic movie recommendation datasets.
- Amazon Beauty & Amazon Office Products: E-commerce datasets containing product reviews and metadata.
Pointwise labels were created by binarizing ratings (e.g., rating $\geq$ 3 is a 1), and pairwise labels were generated by comparing relative ratings of item pairs for the same user.
Key Insights and Ideas
- Momentum as a Denoiser: While momentum is popular in contrastive learning (like MoCo) for maintaining a large dictionary of samples, here it serves a different purpose: acting as a temporal filter to smooth out the "optimization noise" caused by conflicting labels.
- Uncertainty via Divergence: The idea that the gap between a "fast" model and a "slow" model represents uncertainty is a powerful heuristic. It allows for automated sample weighting without needing complex Bayesian machinery.
- Complementary Strengths: Pointwise learning is good for capturing the absolute magnitude of interest, while pairwise learning is superior for ranking. MP2 proves that a structural combination of both—rather than just a mixed loss function—is key to state-of-the-art results.
Key Results
MP2 consistently outperformed competitive baselines like NeuMF (pointwise) and BPR (pairwise) across all datasets.
- Significant Gains: On MovieLens-1m, MP2 achieved an NDCG@20 of 0.5314, significantly higher than NeuMF (0.3002) and BPR (0.5241).
- Sensitivity to Momentum: The experiments showed that a high momentum coefficient (around $\alpha = 0.999$) is optimal. This confirms that a very "slow" and stable item representation is necessary to combat annotation bias.
Limitations and Future Work
While MP2 shows strong results, the authors note a few areas for growth:
- Listwise Learning: The current framework only considers pairs. Moving to listwise learning (ranking a whole set of items at once) could further improve results.
- Computational Overhead: Adding a third tower increases the parameter count and memory footprint, though the momentum update itself is computationally cheap compared to backpropagation.
- Cold Start: The paper doesn't explicitly discuss how the momentum approach handles extremely sparse "cold-start" items where representations might not have enough updates to stabilize.
Summary updated