The trade-offs of model size in large recommendation models : A 10000 × compressed criteo-tb DLRM model (100 GB parameters to mere 10MB)

Aditya Desai, Anshumali ShrivastavaarXiv 2022 · arXiv · OpenReview

Abstract

Embedding tables dominate industrial-scale recommendation model sizes, using up to terabytes of memory. A popular and the largest publicly available machine learning MLPerf benchmark on recommendation data is a Deep Learning Recommendation Model (DLRM) trained on a terabyte of click-through data. It contains 100GB of embedding memory (25+ Billion parameters). DLRMs, due to their sheer size and the associated volume of data, face difficulty in training, deploying for inference, and memory bottlenecks due to large embedding tables. This paper analyzes and extensively evaluates a generic parameter sharing setup (PSS) for compressing DLRM models. We show theoretical upper bounds on the learnable memory requirements for achieving $(1 \pm \epsilon)$ approximations to the embedding table. Our bounds indicate exponentially fewer parameters suffice for good accuracy. To this end, we demonstrate a PSS DLRM reaching 10,000\times compression on Criteo-TB without losing quality. Such a compression, however, comes with a caveat: it requires 4.5\times more iterations to reach the same saturation quality. The paper argues that this tradeoff needs more investigations as it might be significantly favorable. Leveraging the small size of the compressed model, we show a 4.3\times improvement in training latency leading to similar overall training times. Thus, in the tradeoff between system advantage of a small DLRM model vs. slower convergence, we show that scales are tipped towards having a smaller DLRM model, leading to faster inference, easier deployment, and similar training times.

Introduction

In modern machine learning, recommendation systems are the primary drivers of model size. Unlike computer vision or natural language processing models, the scale of recommendation models is dictated by embedding tables. These tables map millions of categorical features—such as User IDs or Product IDs—into dense numerical vectors.

For industrial datasets like Criteo-TB, these tables can exceed 100GB, creating significant engineering challenges:

  1. Memory Bottlenecks: Models often exceed the memory capacity of a single GPU, necessitating complex distributed training across multiple nodes.
  2. Latency: Data retrieval from massive tables spread across a network is inherently slow.
  3. Deployment Constraints: Large models are difficult to deploy on edge devices or small servers.

This research investigates the inherent "over-parameterization" of these tables, exploring how much compression is possible before model performance degrades.

Methodology: Parameter Sharing and ROBE

The core approach utilizes a Parameter Sharing Setup (PSS). Instead of a standard one-to-one mapping where every unique ID has a dedicated row in a massive table, PSS employs a "recovery function" to map multiple IDs to a significantly smaller set of shared weights.

Key technical innovations include:

  • Theoretical Foundations: Using Johnson-Lindenstrauss Transforms (JLT), the authors prove that memory requirements only need to grow logarithmically ($O(\log n)$) with the number of items to effectively approximate the original embedding table.
  • ROBE-style Hashing: The practical implementation uses "Random Offset Block Embedding" (ROBE). By hashing chunks of embedding vectors rather than individual elements, the method aligns with GPU architecture, improving cache efficiency and hardware utilization.
  • Dense Gradient Updates: In highly compressed regimes, the authors found that updating the entire small shared table (dense gradients) is more efficient than sparse updates. The reduced memory footprint allows for extremely fast "scatter" operations on modern hardware.

Results and Evaluation

The method was evaluated on two industry-standard benchmarks: Criteo-Kaggle (~2GB original) and Criteo-TB (100GB original, 25+ billion parameters).

Comparison of ROBE with other compression methods across different datasets.

As shown in the comparative analysis above, ROBE achieves significantly higher compression ratios (up to 10,000× on Criteo-TB) while maintaining or even improving model quality compared to existing techniques like HashingTrick or QR Trick.

Key performance metrics include:

  • 10,000× Compression: On Criteo-TB, the embedding table was reduced from 100GB to 10MB without loss in AUC (Area Under Curve).
  • Inference Speedup: Inference time on the Criteo-TB test set was reduced from 203 seconds to 97 seconds on a single GPU, primarily due to the elimination of inter-GPU communication.
  • Hardware Efficiency: The compressed model achieved a 4.37× improvement in training latency per iteration.

Discussion: The Latency-Convergence Trade-off

A critical insight of this work is the "Latency-Convergence Seesaw." While a 10,000× compressed model requires approximately 4.5× more iterations to reach target accuracy—due to parameters being "stretched" across more IDs—this is offset by the system advantages of a smaller model.

The trade-offs between hardware hierarchy, training time, and convergence iterations.

By shrinking the model to fit within the fastest levels of GPU memory (e.g., GPU cache or global memory instead of multi-node RAM), the time per iteration drops significantly. This "System Advantage" nearly perfectly cancels out the "Convergence Disadvantage," resulting in a total training time that is only 1.04× that of the uncompressed model.

Conclusion

The research demonstrates that the scales are tipped in favor of smaller DLRM models. The massive reduction in size enables faster inference and easier deployment while maintaining competitive training times. Future work may explore how these benefits scale to even larger (50TB+) production models and dynamic environments with evolving feature sets.

Summary updated

Abstract

Embedding tables dominate industrial-scale recommendation model sizes, using up to terabytes of memory. A popular and the largest publicly available machine learning MLPerf benchmark on recommendation data is a Deep Learning Recommendation Model (DLRM) trained on a terabyte of click-through data. It contains 100GB of embedding memory (25+Billion parameters). DLRMs, due to their sheer size and the associated volume of data, face difficulty in training, deploying for inference, and memory bottlenecks due to large embedding tables. This paper analyzes and extensively evaluates a generic parameter sharing setup (PSS) for compressing DLRM models. We show theoretical upper bounds on the learnable memory requirements for achieving $(1 \pm \epsilon)$ approximations to the embedding table. Our bounds indicate exponentially fewer parameters suffice for good accuracy. To this end, we demonstrate a PSS DLRM reaching 10000$\times$ compression on criteo-tb without losing quality. Such a compression, however, comes with a caveat. It requires 4.5 $\times$ more iterations to reach the same saturation quality. The paper argues that this tradeoff needs more investigations as it might be significantly favorable. Leveraging the small size of the compressed model, we show a 4.3$\times$ improvement in training latency leading to similar overall training times. Thus, in the tradeoff between system advantage of a small DLRM model vs. slower convergence, we show that scales are tipped towards having a smaller DLRM model, leading to faster inference, easier deployment, and similar training times.
BibTeX
@misc{desai2022trade,
  author = {Aditya Desai and Anshumali Shrivastava},
  title = {{The trade-offs of model size in large recommendation models : A 10000 × compressed criteo-tb DLRM model (100 GB parameters to mere 10MB)}},
  year = {2022},
  eprint = {2207.10731},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2207.10731},
}
Figure