Scaling Recommender Systems Beyond the Embedding Memory Wall

17 concepts19 papers1 proposal

In deep learning, we often equate model scale with computational intensity. Large Language Models (LLMs), for instance, are primarily compute-bound: their performance is limited by how fast a hardware accelerator can perform dense matrix multiplications (FLOPs).

Deep Learning Recommendation Models (DLRMs) operate under an entirely different paradigm. As highlighted by Hsia et al., recommendation models consume a massive share of industrial AI inference cycles, but their primary bottleneck is not compute—it is memory capacity and bandwidth.

This bottleneck is driven by the embedding table. To process categorical features with high cardinalities (such as billions of unique users or items), recommendation systems map discrete IDs to dense, continuous vectors. As these tables scale to accommodate massive user bases, they balloon into hundreds of gigabytes or even terabytes. This creates a "memory wall": the tables become too large to fit into the fast, on-chip memory—such as GPU High Bandwidth Memory (HBM) or CPU Last Level Cache (LLC)—of a single hardware accelerator.

Traditional Lookup (Memory-Bound):
[Sparse User/Item IDs] ──> [Terabyte-Scale Embedding Table (RAM/SSD)] ──> [High Read Latency] ──> [Compute Layers]

Compressed/Co-Designed Alternative:
[Sparse User/Item IDs] ──> [Compact Representation (Sparsity/Decomposition/Cache)] ──> [SRAM/LLC/HBM] ──> [Fast Inference]

Retrieving these vectors requires frequent, random memory accesses (gather operations). Without spatial locality, the processor spends most of its time stalling, waiting for data to arrive from slower system RAM or distributed networks (Desai et al.; Acun et al.). To scale recommender systems, researchers are attacking this memory wall from three main angles: exploiting structural sparsity, redesigning continuous embeddings into discrete representations, and co-designing hardware-software memory routing.

Exploiting Redundancy via Sparsity and Pruning

Why are embedding tables so massive? Intuitively, they treat every entity as entirely independent, allocating the same number of parameters to a rare item seen once a year as to a viral video viewed millions of times. This leads to extreme over-parameterization.

To prove this redundancy, Wang et al. validated the Lottery Ticket Hypothesis (LTH) within media recommender systems. They demonstrated that massive embedding tables contain tiny, sparse "winning tickets"—sub-networks comprising as little as 3% of the original parameters that can match or even outperform the accuracy of the full model.

Instead of training a gargantuan model and pruning it post-hoc, recent paradigms aim to make models "born sparse." Wang et al. introduced Dynamic Sparse Learning (DSL), a training paradigm that periodically prunes unimportant weights and regrows promising ones within a single training run, slashing storage and inference costs by up to 75%.

Another approach to sparsity is projection. Rather than storing dense float arrays, Kasalický et al. developed CompresSAE, a sparse autoencoder that projects dense embeddings into a high-dimensional but sparsely activated space. Because only a fraction of the dimensions are active for any given item, the system achieves a 12x compression ratio by storing and operating only on non-zero activations.

Factorization and Compositional Codebooks

Instead of allocating a unique, dedicated row in a matrix for every single ID, can we construct an entity's vector on the fly from smaller, shared components?

One mathematical solution is matrix decomposition. Acun et al. introduced TT-Rec, which uses Tensor Train decomposition to compress embedding tables by over 100x. Instead of storing a massive 2D matrix, TT-Rec represents the table as a sequence of low-rank 3D tensors. When an ID lookup occurs, the system performs a sequence of small tensor contractions to reconstruct the vector, trading a small amount of compute for a massive reduction in memory footprint.

Tensor Train Lookup:
ID ──> [Index Mapping] ──> [Core Tensor 1] × [Core Tensor 2] × [Core Tensor 3] ──> Reconstructed Embedding Vector

Alternatively, we can use multiple small "codebooks" that combine compositionally. Chen et al. developed CERP (Compositional Embedding with Regularized Pruning), which uses two compact codebooks to represent users and items. By applying a regularized pruning mechanism, the authors ensure that the codebooks store complementary features. An entity's final embedding is generated by combining entries from both codebooks, allowing the model to represent millions of unique combinations with minimal storage overhead.

Shifting to Discrete and Clustered Representations

If two users share highly similar tastes, giving them completely independent embedding vectors is statistically and structurally wasteful. A third branch of research replaces continuous, high-dimensional tables with discrete or shared representations.

Chen et al. proposed Clustered Embedding Learning (CEL), a framework that dynamically groups users and items into clusters that share a single embedding vector. By using a gradient-based splitting technique, the model refines these clusters over time, achieving up to a 2,650x reduction in memory while simultaneously improving recommendations for "cold-start" entities by allowing them to inherit the robust representations of their cluster.

Taking discretization further, Liu et al. introduced a "semantic-token" paradigm (UIST) for click-through rate prediction. Instead of continuous vectors, it compresses user and item profiles into a few discrete, hierarchical tokens. This 200-fold reduction in memory usage replaces continuous floating-point lookups with discrete token matching, bypassing traditional embedding tables entirely.

The most radical departure from traditional tables is Mem-Rec, proposed by Jha et al.. The authors replaced gigabyte-scale embedding tables entirely with a dual-Bloom filter encoding scheme. Because Bloom filters are compact, probabilistic bit arrays, they can map features directly to classifications within a tiny memory footprint. This allows the entire model to fit inside a CPU’s last-level cache (LLC), accelerating inference speeds by 3.4× by eliminating system RAM access latency altogether.

Hardware-Software Co-Design for Tiered Memory

When embedding tables cannot be compressed enough to fit into fast memory, systems must optimize how data flows across the physical hardware hierarchy.

Because recommendation data access patterns are highly skewed—a tiny fraction of popular items receive the vast majority of interactions—we can separate "hot" and "cold" data. Sethi et al. exploited this with RecShard, a data-driven partitioning and placement strategy for tiered memory systems. RecShard identifies frequently accessed rows and maps them to high-bandwidth GPU memory, while sharding the massive tail of "cold" rows across slower system memory, maximizing training throughput and load balancing.

Tiered Memory Sharding (RecShard):
[Incoming Lookups] 
       │
       ├──> [Hot Rows (High Frequency)]  ──> High-Bandwidth GPU Memory (HBM)
       └──> [Cold Rows (Low Frequency)]  ──> Slower System RAM / SSD

Other approaches focus on accelerating the underlying hardware operations or optimizing cache locality:

  • FPGA Acceleration: Jiang et al. introduced MicroRec, an inference engine that pairs hardware acceleration on FPGAs with custom data structures to handle the high volume of random memory accesses, drastically reducing read latency.
  • CPU Cache Optimization: For environments where GPUs are too costly, Zhang et al. designed HEAT (Highly Efficient and Affordable Training). HEAT optimizes memory locality on multi-core CPUs, structuring data access to eliminate redundant computations and minimize cache misses during embedding retrieval.
  • Quantization and Pruning: In industrial settings, combining multiple techniques is common. For example, Zhang et al. introduced SHARK, which pairs a Taylor-expansion-based feature pruning method with a frequency-aware, row-wise quantization strategy to achieve a 70% storage reduction and a 30% boost in query speed in production.

By rethinking the embedding layer—whether through algorithmic sparsity, mathematical factorization, discrete tokenization, or hardware-aware sharding—modern recommendation architectures are successfully breaking through the memory wall, proving that models do not need to be massive to be powerful.

Key Foundational Work

The trade-offs of model size in large recommendation models : 100GB to 10MB Criteo-tb DLRM model (2022) — contributed 1 concept elements to this direction.
Accelerating Recommendation System Training by Leveraging Popular Choices (2021) — contributed 1 concept elements to this direction.
TT-Rec: Tensor Train Compression for Deep Learning Recommendation Models (2021) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_embedding_table_memory_bottleneck_embedd_ff77bd78,
  title = {embedding table memory bottleneck (embedding table scalability)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/embedding-table-memory-bottleneck-embedding-table-scalability/rdr_ff77bd78350649b2be57b68039b187b7},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_embedding_table_memory_bottleneck_embedd_ff77bd78_v1,
  title = {embedding table memory bottleneck (embedding table scalability)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/embedding-table-memory-bottleneck-embedding-table-scalability/rdr_ff77bd78350649b2be57b68039b187b7/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake