The trade-offs of model size in large recommendation models : 100GB to 10MB Criteo-tb DLRM model

Aditya Desai, Anshumali ShrivastavaNeurIPS 2022 · OpenReview

TL;DR

Researchers from Rice University have demonstrated that massive Deep Learning Recommendation Models (DLRMs) can be compressed by a staggering 10,000x—shrinking from 100GB down to just 10MB—without losing prediction quality. While these tiny models require more iterations to converge, their significantly lower system latency means they can be trained in roughly the same total time as their massive counterparts, all while offering much faster inference and vastly easier deployment.

The Problem: The Memory Wall in Recommendation Systems

In the world of industrial machine learning, recommendation systems are the heavyweights. Unlike language models that focus primarily on dense compute, recommendation models are dominated by embedding tables. These are massive look-up structures that map millions of categorical values (like user IDs, product tags, or locations) into dense vectors.

On benchmarks like Criteo-tb, these tables can exceed 100GB; in industry, they frequently reach tens of terabytes. This "memory wall" creates a nightmare for engineers:

  1. Deployment: You cannot fit a 100GB model on a single standard GPU.
  2. Training: Massive models require complex distributed systems, leading to high communication overhead.
  3. Inference: Looking up values in giant tables across a network is slow and memory-intensive.

The authors tackle a fundamental question: Do we actually need all those parameters, or is there a way to represent this information in a fraction of the space?

Methods: Parameter-Sharing Setup (PSS)

The paper focuses on a Parameter-Sharing Setup (PSS). Instead of each categorical value having its own unique row in a giant table, multiple values share a smaller pool of learnable weights through a mathematical mapping.

Key technical components include:

  • Theoretical Framework: The authors use Johnson-Lindenstrauss Transforms (JLT)—a technique from dimensionality reduction—to prove that you only need a logarithmic number of parameters relative to the number of categories to maintain high-quality approximations.
  • Macro and Micro PSS: They define two ways to measure approximation quality: "Macro" (preserving the geometry of the entire table) and "Micro" (ensuring individual important embeddings remain accurate).
  • Implementation Optimizations: To make the compressed models fast in practice, they implemented hashing chunks (grouping elements to improve cache efficiency) and utilized dense gradients. While standard DLRMs use sparse gradients to save time, highly compressed models benefit more from treating gradients as dense due to better hardware utilization.

Key Insight: The "Convergence vs. Latency" Trade-off

This is the paper's most counter-intuitive finding. In machine learning, smaller models usually take more "epochs" (passes through the data) to reach a certain accuracy. However, because the 10MB model is so small, it fits entirely within fast memory, allowing the system to process each iteration 4.3x faster.

Trade-offs of model size across different hardware boundaries

The "system advantage" of the small model almost perfectly cancels out the "mathematical disadvantage" of slower convergence. By shrinking the model to fit entirely within a single GPU’s high-speed memory (or even a CPU's cache), you eliminate the massive "all-to-all" communication bottlenecks that plague distributed training of large models.

Key Results

The study evaluates its approach on industry-standard benchmarks, including Criteo-tb (a 100GB model) and Criteo-kaggle.

  • Extreme Compression: The researchers achieved 10,000x compression on the Criteo-tb DLRM model (100GB $\rightarrow$ 10MB) with virtually no loss in AUC (Area Under the Curve) quality.
Model quality across different architectures at varying compression rates
  • Training Parity: While the 10,000x compressed model required 4.5 epochs to reach the target quality (compared to 1 epoch for the original), the total training time was only 1.04x the original due to the vastly increased iteration speed.
  • Inference Speedup: The compressed model achieved a 3x improvement in inference latency.
  • Single-Device Training: The researchers were able to train the entire compressed Criteo-tb model on a single GPU, whereas the original requires a distributed cluster of 8 GPUs just to hold the embedding tables.

Limitations and Future Work

  • Task Specificity: The study focuses on Click-Through Rate (CTR) prediction, a binary classification task. It remains to be seen if these extreme compression rates hold for more complex tasks like ranking or multi-task retrieval.
  • First-Layer Focus: The theoretical analysis primarily evaluates the first layer (the embeddings). Analyzing the impact of compression as it propagates through subsequent deep neural network layers is a necessary next step.
  • Hyperparameter Sensitivity: Highly compressed models may require more careful tuning of learning rates and hashing functions to ensure they don't get stuck in poor local minima during the longer training process.

Summary updated

Abstract

Embedding tables dominate industrial-scale recommendation model sizes, using up to terabytes of memory. A popular and the largest publicly available machine learning MLPerf benchmark on recommendation data is a Deep Learning Recommendation Model (DLRM) trained on a terabyte of click-through data. It contains 100GB of embedding memory (25+Billion parameters). DLRMs, due to their sheer size and the associated volume of data, face difficulty in training, deploying for inference, and memory bottlenecks due to large embedding tables. This paper analyzes and extensively evaluates a generic parameter-sharing setup (PSS) for compressing DLRM models. We show theoretical upper bounds on the learnable memory requirements for achieving approximations to the embedding table. Our bounds indicate exponentially fewer parameters suffice for a good approximation. To this end, we demonstrate a PSS DLRM reaching 10000$\times$ compression on criteo-tb without losing quality. Such a compression, however, comes with a caveat. It requires 4.5 $\times$ more iterations to achieve the same saturation quality. The paper argues that this tradeoff needs more investigation as it might be significantly favorable. Leveraging the small size of the compressed model, we show a 4.3$\times$ improvement in training latency leading to similar overall training times. Thus, in the tradeoff between the system advantage of a small DLRM model vs. slower convergence, we show that scales are tipped towards having a smaller DLRM model, leading to the same quality, faster inference, easier deployment, and similar training times.
BibTeX
@inproceedings{desai2022trade,
  author = {Aditya Desai and Anshumali Shrivastava},
  title = {{The trade-offs of model size in large recommendation models : 100GB to 10MB Criteo-tb DLRM model}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=c9I_NArDIjD},
}
Figure