Adaptive Hyper-parameter Learning for Deep Semantic Retrieval

Mingming Li, Chunyuan Yuan, Huimu Wang, Peng Wang, Jingwei Zhuo, Binbin Wang, Lin Liu, Sulong XuEMNLP 2023 · DOI

Abstract

Researchers from JD.com have developed a "hyper-parameter-free" approach to deep semantic retrieval that adaptively adjusts loss function parameters (margin and temperature) for every query-item pair. By replacing fixed global constants with a heuristic based on batch similarity and adding a symmetric metric learning objective to prevent model collapse, they achieved significant performance gains on a 5-billion-sample industrial dataset and successfully deployed the system to a major e-commerce search engine.

Introduction

In modern e-commerce, semantic retrieval is essential for understanding user intent beyond simple keyword matching. State-of-the-art systems typically utilize dual-encoder architectures (e.g., DSSM or BERT) trained with contrastive losses like margin loss or softmax loss. However, these losses depend on two sensitive hyper-parameters: the margin ($\delta$), which controls the separation between positive and negative items, and the temperature ($\tau$), which adjusts the sharpness of the probability distribution.

Traditionally, these are set as global constants via expensive grid searches. This paper argues that a one-size-fits-all approach is flawed because different queries—ranging from general terms like "shoes" to precise specifications like "Nike Air Max 270 Men's Size 10"—possess varying levels of ambiguity and require distinct learning signals.

Proposed Methods

The authors propose a two-pronged solution to make these hyper-parameters adaptive and "free" from manual tuning.

Adaptive Heuristic Method

Instead of fixed values, the model calculates $\delta$ and $\tau$ on the fly. For a given query-item pair, the margin is determined by the similarity between the positive item and the negative items in the current batch. When a negative item is "hard" (highly similar to the positive one), the model adaptively reduces the margin/temperature to focus the learning signal. A stop-gradient operation is employed to ensure that the adaptive margin itself does not destabilize the optimization process.

Symmetric Metric Learning

A significant risk in adaptive learning is "model collapse," where embeddings cluster into a single point to minimize loss. To counter this, the authors introduce a symmetric loss term. While standard retrieval treats the query as the anchor to find items, this term treats the item as the anchor to find the query. This dual-view alignment ensures the embedding space remains well-distributed.

Ablation study showing the critical role of symmetric loss in preventing model collapse during adaptive learning.

Experimental Evaluation

The study utilizes a massive industrial dataset from JD.com search logs, comprising 5 billion samples over 60 days. Evaluation includes offline metrics such as Recall@K and batch-top@K, alongside online A/B testing tracking conversion rates (UCVR) and revenue per visitor (UV-value).

Key Results

The proposed method was tested against strong baselines, including DSSM, DPSR, and RSR. Using a BERT backbone, the adaptive method significantly outperformed existing models.

Comparison of the proposed adaptive method against state-of-the-art baselines on the JD.com dataset.
  • Offline Performance: The adaptive method improved Recall@1 from 0.0094 to 0.0137 (a ~45% relative increase) and showed consistent gains across Recall@50, 500, and 1000.
  • Sensitivity Analysis: The authors demonstrated that even minor changes in fixed margins (e.g., 0.1 to 0.5) can drastically degrade performance, highlighting the robustness of the adaptive approach.
  • Online Impact: In live A/B tests, the method increased the number of items passing to the ranking phase by 2.0% and improved the conversion rate (UCVR) by 0.45%.

Key Insights and Discussion

  • Hardness Correlation: Optimal margins are inversely related to the "hardness" of negative samples. By tying hyper-parameters to batch similarities, the model automatically prioritizes difficult distinctions.
  • Symmetry as a Regularizer: Symmetric loss is not merely a performance booster but a necessary safeguard. Without it, adaptive components are prone to collapsing the metric space, as shown in the ablation studies.
  • Agnosticism: The method succeeds in retrieval because it does not rely on pre-trained user/item biases, but rather on immediate geometric relationships within the training batch.

Conclusion

This research presents a scalable, adaptive framework for semantic retrieval that eliminates the need for manual hyper-parameter tuning while improving both offline accuracy and online business metrics. Future work will explore scaling this logic beyond batch-level similarities and integrating it with multi-objective learning for industrial ranking systems.

Summary updated

BibTeX
@inproceedings{li2023adaptive,
  author = {Mingming Li and Chunyuan Yuan and Huimu Wang and Peng Wang and Jingwei Zhuo and Binbin Wang and Lin Liu and Sulong Xu},
  title = {{Adaptive Hyper-parameter Learning for Deep Semantic Retrieval}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-industry.72},
}
Figure