Infinite Recommendation Networks: A Data-Centric Approach

Noveen Sachdeva, Mehak Preet Dhaliwal, Carole-Jean Wu, Julian J. McAuley

TL;DR

This paper introduces $\infty$-AE, an autoencoder with infinitely wide bottleneck layers that achieves state-of-the-art recommendation performance with a simple closed-form solution. By leveraging this model's efficiency, the authors also develop DISTILL-CF, a framework that compresses massive interaction datasets into tiny, high-fidelity summaries (as small as 0.1% of the original size) that actually improve model performance by denoising the data.

The Problem: Is More Data Always Better?

Modern recommender systems (RecSys) are caught in a "more is better" arms race, training increasingly complex models on billions of user-item interactions. This leads to massive computational costs, significant carbon footprints, and privacy risks.

Furthermore, RecSys data is notoriously "messy"—it is extremely sparse, heterogeneous, and often contains noise from accidental clicks or biased feedback. While "data distillation" (creating small synthetic datasets that represent the whole) has worked for continuous data like images, it has struggled with the discrete, semi-structured nature of recommendation logs. The authors ask a counter-intuitive question: Is more data really what we need, or do we just need better data?

Methodology

The authors propose a two-pronged solution focusing on both model architecture and data quality:

1. $\infty$-AE (Infinite-width Autoencoder)

Drawing on Neural Tangent Kernel (NTK) theory, the authors show that training an infinitely wide autoencoder for an infinite number of steps is equivalent to performing Kernelized Ridge Regression (KRR). This results in a model that is highly expressive yet mathematically simple, featuring a closed-form solution and only a single hyperparameter ($\lambda$).

2. DISTILL-CF

This is a data distillation framework designed specifically for collaborative filtering. It uses a bilevel optimization strategy:

  • Inner Loop: Quickly trains the $\infty$-AE on a small "support set" of synthetic users.
  • Outer Loop: Updates the synthetic users to minimize the reconstruction loss on the original, large dataset.

To handle the discrete nature of user-item interactions, they use differentiable Gumbel-sampling, allowing the model to learn which interactions are most "informative" to include in the synthetic summary.

Key Insights

  • Infinite Beats Finite: While infinite-width networks often underperform their finite counterparts in computer vision, this paper shows the opposite is true for RecSys. A simple, wide, linear-acting network can outperform complex deep learning models like LightGCN.
  • The Denoising Effect: One of the most surprising findings is that models trained on distilled data can actually outperform models trained on the full original dataset. This suggests that DISTILL-CF acts as a filter, stripping away "noisy" interactions and keeping only the core signal of user preferences.
  • Transferability: Data distilled using the $\infty$-AE isn't just useful for that specific model. These tiny synthetic datasets also work remarkably well for training other popular models like EASE and MVAE.

Results

The models were evaluated on four diverse recommendation benchmarks: Amazon Magazine, MovieLens-1M (ML-1M), Douban, and Netflix. The findings challenge the "Big Data" status quo in RecSys.

Performance: $\infty$-AE consistently outperformed state-of-the-art models like LightGCN, EASE, and MVAE across all datasets.

Comparison of $\infty$-AE with SoTA methods. Bold indicates best performance.

Efficiency and Scalability: Using DISTILL-CF, the authors achieved 96–105% of the full-dataset performance using only 0.1% of the original data on the massive Netflix dataset. While the $O(U^2)$ complexity of $\infty$-AE makes it hard to scale to millions of users directly, training it on a distilled set of just 500 "fake users" allows it to reach SoTA performance efficiently.

Performance vs. sampling budget across datasets. DISTILL-CF (purple) saturates much faster than random or heuristic sampling.

Robustness: DISTILL-CF is significantly more robust to noise than standard sampling methods. Even when 5% noise was injected into the original data, the distilled summaries maintained high performance.

Limitations and Future Work

  • Static vs. Dynamic: The current work focuses on static recommendation. Extending these "infinite" models to sequential or temporal recommendation remains an open challenge.
  • Computational Bottleneck: Although distillation reduces the data size, the initial distillation process itself requires an expensive outer-loop optimization.
  • Privacy: While synthetic data summaries could theoretically improve privacy, formal differential privacy guarantees for this specific distillation method haven't been proven yet.

The paper concludes that the RecSys community should shift its focus toward Data-Centric AI—prioritizing the quality and "density" of information over the sheer volume of raw logs.

Summary updated

Abstract

We leverage the Neural Tangent Kernel and its equivalence to training infinitely-wide neural networks to devise $\infty$-AE: an autoencoder with infinitely-wide bottleneck layers. The outcome is a highly expressive yet simplistic recommendation model with a single hyper-parameter and a closed-form solution. Leveraging $\infty$-AE's simplicity, we also develop Distill-CF for synthesizing tiny, high-fidelity data summaries which distill the most important knowledge from the extremely large and sparse user-item interaction matrix for efficient and accurate subsequent data-usage like model training, inference, architecture search, etc. This takes a data-centric approach to recommendation, where we aim to improve the quality of logged user-feedback data for subsequent modeling, independent of the learning algorithm. We particularly utilize the concept of differentiable Gumbel-sampling to handle the inherent data heterogeneity, sparsity, and semi-structuredness, while being scalable to datasets with hundreds of millions of user-item interactions. Both of our proposed approaches significantly outperform their respective state-of-the-art and when used together, we observe $96-105$% of $\infty$-AE's performance on the full dataset with as little as $0.1$% of the original dataset size, leading us to explore the counter-intuitive question: Is more data what you need for better recommendation?
BibTeX
@inproceedings{sachdeva2022infinite,
  author = {Noveen Sachdeva and Mehak Preet Dhaliwal and Carole-Jean Wu and Julian J. McAuley},
  title = {{Infinite Recommendation Networks: A Data-Centric Approach}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=xRqh5Zpx936},
}
Figure