Harmless Transfer Learning for Item Embeddings

Chengyue Gong, Xiaocong Du, Dhruv Choudhary, Bhargav Bhushanam, Qiang Liu, Arun KejariwalNAACL 2022 · OpenReview

Introduction

In Natural Language Processing (NLP) and Recommendation Systems (RecSys), data typically follows a "long-tail" distribution. While a few "head" items appear frequently, the vast majority of "tail" items are rare. Standard deep learning models often fail to learn meaningful embeddings for these rare items due to data scarcity.

To address this, existing solutions often employ Transfer Learning, using an auxiliary "transfer loss" to align the embeddings of rare items with those of frequent ones. However, this approach relies on the risky assumption that rare items should inherently resemble frequent ones. When this assumption is incorrect, transfer learning introduces "harmful bias," degrading the model's overall accuracy. This paper introduces a "Harmless Transfer Learning" framework designed to boost the performance of rare items without compromising the quality of frequent ones.

The Harmless Transfer Learning Framework

The proposed framework achieves "harmlessness" through two primary mechanisms: selective filtering and lexicographic optimization.

Overview of the Harmless Transfer Learning framework, featuring selective transfer and adaptive coefficient control.

1. Selective Transfer Loss

Rather than applying transfer learning indiscriminately, the model uses co-occurrence data to identify which rare items are genuinely related to frequent ones. If a rare item lacks a contextual relationship with a frequent item, it is excluded from the transfer process. This ensures that knowledge is shared only between items likely to possess semantic or functional similarities.

2. Lexicographic (Lexico) Optimization

Standard models typically minimize a weighted sum of the primary loss and the transfer loss ($\ell_{main} + \lambda \ell_{transfer}$). Finding the optimal $\lambda$ usually requires exhaustive grid searches. The authors propose a hierarchical approach: minimize the transfer loss only to the extent that it does not interfere with the primary task. By adapting the Dynamic Barrier Gradient Descent (DBGD) algorithm, the model calculates an item-wise, adaptive weight ($\lambda_i$) at every training step, ensuring the primary task remains the priority.

Experimental Evaluation

The framework was evaluated across diverse domains, including Language Modeling (Wikitext-2, Wikitext-103), Named Entity Recognition (CoNLL-03), Machine Translation (WMT2014 EN-DE), and Recommendation Systems (MovieLens 1M, Criteo).

Key Results

The "Harmless" approach consistently outperformed both baseline models and traditional transfer learning methods:

  • NLP Performance: On the Wikitext-103 dataset, the model significantly reduced "tail" word perplexity from 313.2 to 275.6, while also yielding slight improvements for "head" words.
  • RecSys Performance: On the Criteo dataset, which features an extreme long tail, the method achieved a significant boost in Area Under the Curve (AUC) across various model architectures.
  • Robustness: The lexicographic approach eliminates the need for manual hyperparameter tuning for $\lambda$, achieving superior results automatically.
Comparison of test perplexity on WT2 and WT103 datasets, showing the proposed method outperforming linear weighting.

Discussion and Insights

A core insight of this work is that the "similarity assumption" often found in transfer learning can be a trap. Forcing similarity between head and tail distributions when they are fundamentally different leads to performance degradation. By operating at an item-wise granularity, the model can be aggressive with transfer for some items while ignoring it for others.

Furthermore, the framework is designed for plug-and-play compatibility. It acts as a training-time module that can be integrated into existing pipelines, such as Transformers or DeepFM, with minimal computational overhead (approximately 5-10%).

Conclusion and Future Work

While the current selective strategy relies on a pre-calculated co-occurrence matrix, future work could involve developing an online version that updates the target domain dynamically during training. Additionally, while this paper focuses on item embeddings, the "harmless" philosophy could potentially be extended to other multi-task or transfer learning scenarios. The authors also aim to scale this approach to even larger industrial datasets in the future.

Summary updated

BibTeX
@inproceedings{gong2022harmless,
  author = {Chengyue Gong and Xiaocong Du and Dhruv Choudhary and Bhargav Bhushanam and Qiang Liu and Arun Kejariwal},
  title = {{Harmless Transfer Learning for Item Embeddings}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=rn8GQlZ97sS},
}
Figure