TL;DR
This paper introduces $\infty$-AE, an autoencoder with infinitely wide bottleneck layers that achieves state-of-the-art recommendation performance with a simple closed-form solution. By leveraging this model's efficiency, the authors also develop DISTILL-CF, a framework that compresses massive interaction datasets into tiny, high-fidelity summaries (as small as 0.1% of the original size) that actually improve model performance by denoising the data.
The Problem: Is More Data Always Better?
Modern recommender systems (RecSys) are caught in a "more is better" arms race, training increasingly complex models on billions of user-item interactions. This leads to massive computational costs, significant carbon footprints, and privacy risks.
Furthermore, RecSys data is notoriously "messy"—it is extremely sparse, heterogeneous, and often contains noise from accidental clicks or biased feedback. While "data distillation" (creating small synthetic datasets that represent the whole) has worked for continuous data like images, it has struggled with the discrete, semi-structured nature of recommendation logs. The authors ask a counter-intuitive question: Is more data really what we need, or do we just need better data?
Methodology
The authors propose a two-pronged solution focusing on both model architecture and data quality:
1. $\infty$-AE (Infinite-width Autoencoder)
Drawing on Neural Tangent Kernel (NTK) theory, the authors show that training an infinitely wide autoencoder for an infinite number of steps is equivalent to performing Kernelized Ridge Regression (KRR). This results in a model that is highly expressive yet mathematically simple, featuring a closed-form solution and only a single hyperparameter ($\lambda$).
2. DISTILL-CF
This is a data distillation framework designed specifically for collaborative filtering. It uses a bilevel optimization strategy:
- Inner Loop: Quickly trains the $\infty$-AE on a small "support set" of synthetic users.
- Outer Loop: Updates the synthetic users to minimize the reconstruction loss on the original, large dataset.
To handle the discrete nature of user-item interactions, they use differentiable Gumbel-sampling, allowing the model to learn which interactions are most "informative" to include in the synthetic summary.
Key Insights
- Infinite Beats Finite: While infinite-width networks often underperform their finite counterparts in computer vision, this paper shows the opposite is true for RecSys. A simple, wide, linear-acting network can outperform complex deep learning models like LightGCN.
- The Denoising Effect: One of the most surprising findings is that models trained on distilled data can actually outperform models trained on the full original dataset. This suggests that DISTILL-CF acts as a filter, stripping away "noisy" interactions and keeping only the core signal of user preferences.
- Transferability: Data distilled using the $\infty$-AE isn't just useful for that specific model. These tiny synthetic datasets also work remarkably well for training other popular models like EASE and MVAE.
Results
The models were evaluated on four diverse recommendation benchmarks: Amazon Magazine, MovieLens-1M (ML-1M), Douban, and Netflix. The findings challenge the "Big Data" status quo in RecSys.
Performance: $\infty$-AE consistently outperformed state-of-the-art models like LightGCN, EASE, and MVAE across all datasets.
Efficiency and Scalability: Using DISTILL-CF, the authors achieved 96–105% of the full-dataset performance using only 0.1% of the original data on the massive Netflix dataset. While the $O(U^2)$ complexity of $\infty$-AE makes it hard to scale to millions of users directly, training it on a distilled set of just 500 "fake users" allows it to reach SoTA performance efficiently.
Robustness: DISTILL-CF is significantly more robust to noise than standard sampling methods. Even when 5% noise was injected into the original data, the distilled summaries maintained high performance.
Limitations and Future Work
- Static vs. Dynamic: The current work focuses on static recommendation. Extending these "infinite" models to sequential or temporal recommendation remains an open challenge.
- Computational Bottleneck: Although distillation reduces the data size, the initial distillation process itself requires an expensive outer-loop optimization.
- Privacy: While synthetic data summaries could theoretically improve privacy, formal differential privacy guarantees for this specific distillation method haven't been proven yet.
The paper concludes that the RecSys community should shift its focus toward Data-Centric AI—prioritizing the quality and "density" of information over the sheer volume of raw logs.
Summary updated