Learning Effective and Efficient Embedding via an Adaptively-Masked Twins-based Layer

Bencheng Yan, Pengjie Wang, Kai Zhang, Wei Lin, Kuang-Chih Lee, Jian Xu, Bo Zheng

Deep Learning Recommendation Models (DLRMs) are the backbone of modern personalized services, yet they face a significant "memory wall." These models rely on embedding tables to represent categorical features like User IDs or Product IDs as numerical vectors. Traditionally, every ID is assigned a fixed-size vector, leading to massive memory consumption and suboptimal performance due to the "frequency gap"—where high-frequency features are under-represented and low-frequency features are prone to overfitting.

Researchers at Alibaba Group have introduced the Adaptively-Masked Twins-based Layer (AMTL) to address these inefficiencies. AMTL automatically optimizes embedding dimensions for individual features, significantly reducing memory usage while improving predictive accuracy.

The Embedding Dimension Problem

Existing approaches to embedding dimension optimization generally fall into three categories: uniform assignment, rule-based heuristics, and Neural Architecture Search (NAS). As shown in the comparison below, these methods often lack the flexibility to handle the vast scale and dynamic nature of industrial recommendation data.

Comparison of embedding dimension assignment strategies: (a) Standard uniform, (b) Rule-based human design, (c) NAS-based discrete search, and (d) the proposed AMTL with continuous adjustment.

Unlike NAS-based methods that select from a few discrete sizes, AMTL operates in a continuous integer space, allowing for much finer granularity. Crucially, it supports "warm starts," enabling the optimization of pre-trained embedding tables without requiring training from scratch.

The AMTL Architecture

The AMTL layer sits atop a standard embedding layer. Rather than physically resizing vectors during training, it learns to "mask" (zero out) unnecessary dimensions.

Detailed architecture of the Adaptively-Masked Twins-based Layer (AMTL), illustrating the twins-based branches (h-AML, l-AML) and the differentiable masking mechanism.

Key Technical Components:

  • Twins-based Architecture: To prevent high-frequency features from dominating the learning process, AMTL uses two branches: h-AML (high-frequency) and l-AML (low-frequency). A weighted sum combines their outputs, ensuring rare features receive an unbiased dimension selection.
  • Masking Mechanism: The layer generates a mask vector $m_i$. If the model determines a feature only requires a subset of dimensions, the mask zeros out the remainder. In production, these zeroed dimensions are pruned to save space.
  • Differentiable Learning: To make the discrete operation of dimension selection trainable via gradient descent, the authors employ a Tempered Softmax and the Straight-Through Estimator (STE).

Experimental Results

The authors evaluated AMTL on three datasets: MovieLens (1M records), IJCAI-AAC (478k records), and a massive Taobao industrial dataset containing 50 billion records.

  • Accuracy Gains: AMTL consistently outperformed standard fixed embeddings (FBE), rule-based methods (MDE), and NAS-based methods (AutoEmb) in terms of Area Under the Curve (AUC).
  • Memory Efficiency: On the Taobao dataset, AMTL reduced the average embedding dimension from 300 to 110—a 63.3% reduction in memory—with no loss in predictive power.
  • Practical Flexibility: In "warm start" scenarios, AMTL provided a 1.6% AUC gain over the best baseline, whereas other methods often performed worse than standard embeddings when loading pre-trained weights.

Limitations and Future Work

While AMTL offers significant benefits, it introduces a slight training overhead, increasing time per epoch by approximately 9%. Additionally, the model is sensitive to hyperparameters like the softmax "temperature." Future research may focus on scaling AMTL to handle hundreds of feature fields simultaneously without increasing architectural complexity.

Summary updated

BibTeX
@inproceedings{yan2021learning,
  author = {Bencheng Yan and Pengjie Wang and Kai Zhang and Wei Lin and Kuang-Chih Lee and Jian Xu and Bo Zheng},
  title = {{Learning Effective and Efficient Embedding via an Adaptively-Masked Twins-based Layer}},
  booktitle = {ACM International Conference on Information and Knowledge Management},
  year = {2021},
  doi = {10.1145/3459637.3482130},
}
Figure