Domain-specific transformer models for query translation

Mandar Kulkarni, Nikesh Garera, Anusua TrivediACL 2023

TL;DR

This paper tackles the challenge of translating e-commerce search queries that mix English and Hindi (Hinglish), especially when brand names are involved. They introduce a novel "brand loss" during transformer model training to preserve brand names while translating the rest of the query, significantly improving translation accuracy and user experience.

What Problem Does This Paper Address?

The rise of e-commerce has led to a surge in online product listings. In regions like India, a large population uses Hinglish for online searches, creating a significant challenge for search engines. Specifically, when users search for products, they often include brand names. However, many brand names in India are themselves Hinglish words, and directly translating them can lead to incorrect search results. For instance, translating "sasta dabur lal tel" literally might yield "cheap dabur red oil," losing the crucial brand identifier "dabur lal." The core problem is to translate Hinglish queries into English for effective search, while ensuring that brand names are either preserved correctly or corrected if misspelled, without altering their intended meaning.

Methods

The authors propose a modification to the standard Transformer architecture for query translation. Their key innovation is the introduction of a "brand loss" during the training process.

Here's a breakdown of their approach:

  • Brand Name Tokenization: High-demand brand names are added as special tokens to the model's vocabulary. Their embeddings are randomly initialized, allowing the model to learn them from scratch.
  • Brand Loss Function: This is a novel cross-entropy loss computed using a denoising auto-encoder objective specifically on brand name data. This loss encourages the model to correctly reconstruct brand names, even when they are misspelled or part of a Hinglish query.
  • Warm-Starting and Fine-tuning: They leverage pre-trained Transformer models (like BART/T5) and fine-tune them on their domain-specific data.
  • Data Augmentation: To improve robustness, they employ autoencoder and spell augmentation techniques on the query data.
  • Knowledge Distillation: To address latency constraints for production deployment, they use knowledge distillation. A larger, accurate "teacher" model (their fine-tuned BART) is used to generate pseudo-labels for a large unlabeled dataset, which then trains a smaller, faster "student" model (an Open-NMT model).
  • Weight Quantization: Further optimization for the student model is achieved through weight quantization.

Datasets

  • Manually Tagged Dataset: A dataset of approximately 116,000 manually tagged queries, including Hinglish and English queries from various domains (Grocery, fashion, mobile, footwear).
  • Grocery Brand Names List: A list of 2,226 high-demand Grocery brand names, provided by the product team, used for computing the brand loss.
  • Test Dataset: 10,715 manually tagged queries specifically from the Grocery domain.
  • Hindi Unlabeled Query Corpus: A large internal corpus of Hindi unlabeled queries used for pre-training.

Key Insights and Ideas

  • Brand Loss is Key: The most significant insight is that explicitly penalizing incorrect brand name translations (via "brand loss") is more effective than simply including brand names as additional training data. This allows the model to focus on translating context words accurately while preserving brand integrity.
  • Preservation vs. Correction: The model is designed to both preserve correctly spelled brand names and correct misspelled ones, a crucial distinction for e-commerce search.
  • Domain-Specific Adaptation: Fine-tuning general-purpose pre-trained models like BART on domain-specific data with custom loss functions is highly effective.
  • Latency vs. Accuracy Trade-off: Knowledge distillation is a viable strategy to achieve significant latency reductions with only a minor impact on accuracy, making models production-ready.
  • Hinglish Queries are Complex: The paper highlights the inherent difficulty of Hinglish translation due to variations in spelling and the need to distinguish between brand names and common words.

Key Results

  • Improved Accuracy: The proposed model with brand loss achieved a BLEU score of 70.9, outperforming the baseline without brand loss (68.8).
  • Brand Name Preservation: Table 1 demonstrates that the model with brand loss correctly preserves brand names (e.g., "aashirvaad atta," "dabur lal oil") and corrects misspelled ones (e.g., "asirwaad" to "aashirvaad").
  • Latency Reduction: Knowledge distillation with Open-NMT resulted in a ~28x reduction in p95 latency (from ~200ms to ~7ms) with only a 0.2 drop in BLEU score.
  • Production Deployment: The model, after positive A/B testing results, is currently deployed in production and handles a large volume of queries.
  • BART vs. T5: The BART-base model outperformed T5-base and T5-small models, likely due to BART's denoising pre-training objectives aligning better with the proposed brand loss and data augmentation.

Limitations and Future Work

  • Latency vs. Accuracy Trade-off: While knowledge distillation significantly reduces latency, there is a slight drop in accuracy. The authors note that the student model provides "comparable results," implying a minor but acceptable compromise.
  • Future Work: The paper implicitly suggests that further exploration of different distillation techniques or student architectures could potentially close the accuracy gap further. The success of the approach also opens doors for applying similar domain-specific loss functions to other challenging translation tasks.

Summary updated

BibTeX
@inproceedings{kulkarni2023domain,
  author = {Mandar Kulkarni and Nikesh Garera and Anusua Trivedi},
  title = {{Domain-specific transformer models for query translation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure