Infinite Recommendation Networks: A Data-Centric Approach

Noveen Sachdeva, Mehak Preet Dhaliwal, Carole-Jean Wu, Julian J. McAuley

Introduction

Modern recommender systems (RS) are currently caught in a "Big Data" paradox. While training on billions of user-item interactions can improve accuracy, it comes at a significant cost: massive computational requirements, a substantial carbon footprint, and heightened privacy risks. Furthermore, the field remains divided on a fundamental architectural question: do we truly need deep, complex non-linear models, or are wide, shallow linear models sufficient for the sparse, discrete nature of recommendation data?

This research introduces a two-pronged solution to these challenges: $\infty$-AE, a model that leverages the theory of Neural Tangent Kernels (NTK) to simulate an infinitely wide autoencoder, and DISTILL-CF, a data-centric framework that shrinks massive datasets into tiny, high-fidelity summaries. Together, these methods suggest that for recommendation tasks, model width and data quality are more critical than model depth and data quantity.

The Infinite Autoencoder ($\infty$-AE)

Theoretical advancements in computer vision have shown that as neural networks become infinitely wide, their behavior converges to a Gaussian Process, and their training can be described by the Neural Tangent Kernel (NTK). The authors adapt this theory to the recommendation domain, demonstrating that training an infinitely wide autoencoder is mathematically equivalent to performing Kernelized Ridge Regression (KRR).

The Closed-Form Advantage

Unlike standard deep learning models that require iterative training via Stochastic Gradient Descent (SGD), $\infty$-AE offers a closed-form mathematical solution. By solving a single matrix equation, the model achieves the expressivity of an infinite-width network without the architectural complexity. This approach reduces the model to a single primary hyperparameter—the regularization term ($\lambda$)—greatly simplifying the tuning process while maintaining high performance.

DISTILL-CF: Collaborative Filtering Distillation

To address the computational bottleneck of processing massive datasets, the authors developed DISTILL-CF. This method distills a large, sparse interaction matrix into a much smaller set of synthetic "fake users" that capture the essential collaborative filtering patterns of the original population.

Bilevel Optimization and Differentiable Sampling

DISTILL-CF operates through a bilevel optimization framework:

  1. Inner Loop: A model (such as $\infty$-AE) is trained on the synthetic data.
  2. Outer Loop: The performance of that model is evaluated against the real, full dataset, and the synthetic data is updated to minimize the error.

Because recommendation data is discrete (e.g., a user either interacted with an item or they did not), the authors utilized Differentiable Gumbel-Sampling. This technique allows the discrete synthesis process to be differentiable, enabling the system to "learn" the optimal synthetic interactions through standard backpropagation.

Experimental Evaluation

The researchers evaluated their approach on four diverse benchmarks: Amazon Magazine (small/sparse), MovieLens-1M (standard benchmark), Douban (social media), and Netflix (large-scale).

Key Findings

  • The Superiority of Width: In a departure from trends in computer vision, where finite networks often outperform infinite ones, $\infty$-AE consistently beat state-of-the-art finite models like LightGCN, EASE, and MVAE. This suggests that for sparse recommendation data, "width" is a more potent architectural feature than "depth."
  • Extreme Data Compression: DISTILL-CF achieved remarkable efficiency, reaching 96–105% of the original performance using as little as 0.1% of the original data. On the Netflix dataset, $\infty$-AE achieved superior results using a sample of just 5.4% of users.
  • The "Less is More" Paradox: In several experiments, models trained on distilled data actually outperformed those trained on the full dataset. The distillation process acts as a natural denoiser, stripping away accidental or low-signal interactions and leaving only the core preference patterns.
  • Cross-Model Universality: Data distilled using $\infty$-AE is not model-specific; it proved effective for training other popular architectures, including EASE and MVAE, demonstrating that the distilled summaries capture universal collaborative filtering signals.

Robustness and Scalability

One of the most significant advantages of DISTILL-CF is its robustness. When 5% random noise was injected into the MovieLens dataset, DISTILL-CF maintained significantly higher performance than standard sampling methods. By focusing on the most informative patterns, the distillation process effectively "sees through" the noise that typically degrades model performance.

However, $\infty$-AE does face a scalability challenge: the closed-form solution requires inverting a large matrix, which has a computational complexity of approximately $O(U^{2.376})$ where $U$ is the number of users. DISTILL-CF directly mitigates this by reducing the number of users to a tiny fraction of the original, making the kernel-based approach viable even for massive datasets.

Conclusion

This research challenges the "more data is always better" dogma in the recommendation industry. By combining the theoretical power of infinite-width networks with a principled data-centric distillation framework, the authors demonstrate that it is possible to achieve state-of-the-art performance with a fraction of the data and computational overhead. Future work will explore the privacy implications of synthetic data and the application of these distillation techniques to sequential and click-through rate (CTR) prediction tasks.

Summary updated

BibTeX
@inproceedings{sachdeva2022infinite,
  author = {Noveen Sachdeva and Mehak Preet Dhaliwal and Carole-Jean Wu and Julian J. McAuley},
  title = {{Infinite Recommendation Networks: A Data-Centric Approach}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=xRqh5Zpx936},
}
Figure