In deep learning architectures—particularly recommender systems—embedding tables frequently consume the vast majority of a model's memory footprint. Traditionally, these embeddings are allocated uniformly: every entity (whether a user, an item, or a feature) receives a vector of the exact same size, such as 64 or 128 dimensions.
However, real-world interaction data is highly skewed, typically following a power-law distribution. A small fraction of "head" entities possess millions of interactions, while a massive "long tail" of entities appears only a handful of times. This creates a fundamental optimization dilemma. As noted by Guoqiang Zhou et al. in Object-aware Policy Network in Deep Recommender Systems, static embedding sizes force an unacceptable trade-off: large vectors overfit on rare entities due to data sparsity, while small vectors underfit popular entities by failing to capture their nuanced, multimodal patterns.
To resolve this, a powerful paradigm has emerged: Dynamic Embedding Capacity Allocation. Instead of treating embedding tables as static matrices, modern architectures dynamically scale the dimensionality, count, or activation of embedding vectors based on data density and real-time context.
Popularity-Driven Capacity Search
The earliest realizations of this idea focused on global entity statistics, primarily popularity or frequency. The core intuition is straightforward: an entity's representation capacity should be proportional to its information density.
[Rare Item: 3 Views] ──> [Controller / RL] ──> Small Embedding (e.g., 4-dim)
[Popular Item: 1M Views] ──> [Controller / RL] ──> Large Embedding (e.g., 64-dim)
To automate this mapping without tedious manual tuning, Manas Joglekar et al. introduced Neural Input Search (Neural Input Search for Large Scale Recommendation Models). Their framework leverages reinforcement learning to automatically discover optimal multi-size embedding configurations, balancing the representation needs of popular versus rare items within a shared optimization objective.
In production environments, however, popularity is rarely static. A viral video or a seasonal product can cause an item's interaction frequency to spike instantly. To address streaming environments, Xiangyu Zhao 0001 et al. proposed AutoEmb (Automated Embedding Dimensionality Search in Streaming Recommendations). AutoEmb utilizes a continuous controller network that takes an entity's current popularity and contextual features as inputs, outputting a probability distribution over a discrete set of candidate embedding dimensions. This allows the model to dynamically scale an entity's representational capacity up or down in lockstep with its real-time data velocity.
Instance-Wise Gating and Masking
While popularity-based allocation optimizes the model globally, it ignores the immediate context of an individual interaction. A highly active user interacting with a rare item requires a representation tailored to that specific intersection. This realization shifted the field toward instance-wise capacity allocation, where embedding dimensions are masked or activated on the fly for every forward pass.
A common mechanism for achieving this is the use of probabilistic gates. For instance, Yakun Yu et al. developed iHAS (Instance-wise Hierarchical Architecture Search for Deep Learning Recommendation Models), which implements Bernoulli gates paired with a polarization regularizer. For any given user-item interaction, the model predicts a vector of probabilities—one for each embedding dimension—and samples a binary mask from a Bernoulli distribution. The polarization regularizer forces these probabilities toward 0 or 1, ensuring a crisp, sparse selection of dimensions that are actively relevant to the current instance.
Raw Embedding: [ x₁ , x₂ , x₃ , x₄ , ... , xₙ ]
↓ (Element-wise Multiply)
Bernoulli Gate: [ 1 , 0 , 1 , 0 , ... , 1 ]
↓
Gated Embedding: [ x₁ , 0 , x₃ , 0 , ... , xₙ ]
This gating concept extends beyond recommendation models into self-supervised representation learning. In Bayesian Gated Non-Negative Contrastive Learning (BayesNCL), Peng Cui et al. utilize a similar probabilistic gating mechanism to solve the "optimization conflict" in contrastive learning. When two images share a generic background (e.g., a blue sky), standard contrastive models mistakenly align their entire embedding vectors. By predicting activation probabilities for individual feature dimensions, BayesNCL dynamically suppresses shared background noise, isolating only the semantic dimensions critical to the specific object instance.
Instead of masking individual dimensions of a single vector, capacity can also be modulated by changing the number of active vectors. In Prototypical Contrastive Learning and Adaptive Interest Selection for Candidate Generation in Recommendations, Ningning Li et al. present an Adaptive Interest Selection framework. Here, a neural network predicts a binary mask based on user context to dynamically activate a variable number of distinct "interest profiles." Less active users are allocated fewer interest vectors to prevent noise propagation, while highly active users unlock a larger fleet of vectors to map their complex preferences.
Multi-Vector Allocation and Sequential Exploration
As capacity allocation moves from static tables to dynamic policies, two major challenges arise: how to represent highly multifaceted entities, and how to handle the exploration-exploitation trade-off when searching for the right embedding size.
To address multifaceted entities, Shilong Bao et al. proposed DPCML (Improved Diversity-Promoting Collaborative Metric Learning for Recommendation). Rather than squeezing a user's diverse tastes into a single high-dimensional vector, the authors scale the count of independent embedding vectors assigned to a user based on their interaction history. Users with extensive histories are statistically more likely to possess multimodal preferences; DPCML matches this by allocating multiple distinct vectors, ensuring both recommendation accuracy and diversity without wasting parameters on single-interest users.
Finally, when embedding sizes are adjusted dynamically in streaming environments, the system must learn which size works best through trial and error. If the model prematurely locks into a small embedding size for an item, it may never discover that a larger embedding would yield significantly better predictions.
┌──> Choice A: 8-dim (Low Memory, Safe)
Context (User/Item) ───┼──> Choice B: 32-dim (Medium Memory/Accuracy)
└──> Choice C: 64-dim (High Capacity, Risk of Overfit)
↓
[Contextual Bandit Policy] ──> Minimize Regret
To formalize this challenge, Bowei He et al. introduced DESS (Dynamic Embedding Size Search with Minimum Regret for Streaming Recommender System). The authors model embedding size selection as a non-stationary contextual bandit problem. By treating candidate vector sizes as arms in a bandit framework, the system explicitly balances exploration (trying larger or alternative dimensions to see if accuracy improves) with exploitation (using the embedding size that has historically minimized prediction regret). This formulation ensures that the model's internal representation capacity continuously adapts to evolving data streams with mathematical guarantees on efficiency.
Key Foundational Work
AutoEmb: Automated Embedding Dimensionality Search in Streaming Recommendations (2021) — contributed 6 concept elements to this direction.
Neural Input Search for Large Scale Recommendation Models (2020) — contributed 1 concept elements to this direction.