An End-to-End Solution for Named Entity Recognition in eCommerce Search

Xiang Cheng, Mitchell Bowden, Bhushan Ramesh Bhange, Priyanka Goyal, Thomas Packer, Faizan JavedAAAI 2021 · DOI

Presented at AAAI-21 by researchers at The Home Depot, this work introduces an end-to-end system for Named Entity Recognition (NER) specifically tailored for the complex requirements of eCommerce search. By bridging the gap between academic benchmarks and industrial constraints, the team developed a framework that significantly improves product discovery and revenue.

The Challenge: Academic Models vs. Industrial Reality

In eCommerce, understanding a search query like "Milwaukee 18v drill" requires identifying "Milwaukee" as a Brand and "drill" as a Product Type. While academic NER models perform well on clean, standardized benchmarks, they often struggle in production environments due to:

  1. Noisy Data: Queries are rife with typos, abbreviations, and slang.
  2. Scale and Diversity: With over 14,000 brands and 11,000 product types, manual labeling for all combinations is unfeasible.
  3. Production Constraints: Real-time search requires millisecond latency, making many heavy deep learning models impractical.

Methods: The TripleLearn Framework

The core innovation is TripleLearn, an iterative training framework that combines small amounts of high-quality human-labeled data with massive amounts of noisy and synthetic data.

The end-to-end TripleLearn system architecture.

The framework utilizes three distinct datasets:

  • Golden Data: Small, high-quality, human-annotated queries.
  • Large Noisy Data: Millions of queries labeled automatically via rule-based matching.
  • Synthetic Data: Automatically generated queries to ensure 100% coverage of the product catalog.

The model architecture employs a BiGRU-CRF structure. Gated Recurrent Units (GRU) were chosen over LSTMs for their efficiency, having 25% fewer parameters. To handle typos, a BiLSTM subgraph generates character-level embeddings, allowing the model to decipher misspelled words by their internal structure.

Crucially, TripleLearn uses a consensus mechanism: the model only learns from noisy data if its own prediction matches the noisy label. This prevents the model from reinforcing its own errors during the iterative process.

Key Insights

  • The Power of Iteration: Training on all data simultaneously proved less effective than the iterative approach. The model's performance improved steadily as it "filtered" the noisy data over multiple cycles.
  • Character Embeddings: By analyzing individual characters, the model became robust against the "long tail" of search—typos and abbreviations not found in standard vocabularies.
  • BERT Limitations: Standard BERT models, pre-trained on long-form text like Wikipedia, performed worse than the custom RNN on short, fragmented eCommerce queries.
  • CPU Optimization: The model was optimized to run on standard CPUs, achieving a 99th percentile latency of just 26ms.

Key Results and Business Impact

The TripleLearn framework dramatically outperformed the existing infrastructure. The F1 score improved from 69.5 (legacy system) to 93.3.

F1 score improvement across TripleLearn iterations.

By the seventh iteration, the model achieved 100% coverage of all brands and product types in the catalog. In live A/B testing on homedepot.com, these improvements translated to higher click-through rates and better revenue conversion, resulting in an estimated $60 million annual revenue increase.

Future Directions

The authors plan to expand the model to identify additional attributes like color, size, and material. They also aim to automate the "refresh" cycle of TripleLearn to incorporate daily catalog changes and explore using the NER engine to enrich product descriptions and build a company-wide knowledge graph.

Summary updated

BibTeX
@inproceedings{cheng2021end,
  author = {Xiang Cheng and Mitchell Bowden and Bhushan Ramesh Bhange and Priyanka Goyal and Thomas Packer and Faizan Javed},
  title = {{An End-to-End Solution for Named Entity Recognition in eCommerce Search}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i17.17773},
}
Figure