Search Query Refinement for Japanese Named Entity Recognition in E-commerce Domain

Yuki Nakayama, Ryutaro Tatsushima, Erick Mendieta, Koji Murakami, Keiji ShinzatoNAACL 2024 · DOI

TL;DR

This paper introduces a novel two-stage query refinement method for Japanese e-commerce search queries, combining dictionary matching with character-level BERT-CRF. This approach significantly improves the accuracy of Named Entity Recognition (NER) for e-commerce tasks, demonstrating its practical value in real-world applications.

What Problem Does This Paper Address?

In e-commerce, understanding user search queries is crucial for providing relevant results. Named Entity Recognition (NER) plays a key role in identifying important attributes like brand names. However, Japanese search queries are often malformed due to typos or unconventional phrasing, leading to poor NER performance. While query refinement techniques exist for English, there's a significant gap in research for Japanese e-commerce queries, especially concerning their impact on downstream NLP tasks like NER. This paper aims to bridge this gap by proposing and evaluating a robust query refinement system specifically for Japanese e-commerce.

Methods

The authors propose a two-stage query refinement method:

  1. Stage 1: Dictionary Matching: This stage canonicalizes raw queries by matching terms against three dictionaries:
  2. ManDic: A manually curated list of e-commerce specific phrases.
  3. WikiDic: Phrases extracted from Wikipedia articles.
  4. QryDic: Phrases derived from analyzing past e-commerce search queries, using PMI (Pointwise Mutual Information) to identify correlated word sequences. The matching is performed using a leftmost-longest strategy, and characters within matched terms are assigned chunk tags.

  5. Stage 2: Character-level BERT-CRF: For characters not covered by dictionary matching, a character-based BERT-CRF model is employed. This model leverages the hidden representations from a pre-trained character-BERT model. Crucially, it incorporates the results from the dictionary matching stage to enhance the input representations, focusing on matched terms to avoid propagating errors from malformed segments. A Conditional Random Field (CRF) layer is then used to predict the final chunk tag sequence.

Datasets

  • Query Refinement Task Data: Two Japanese annotators labeled 10,209 queries with BILOU chunk tags. The dataset statistics (train, dev, test) are provided in Table 1, detailing the number of queries with over-merged, over-tokenized, and no malformed terms.
  • Brand Extraction Task Data: One additional annotator labeled brand tags for 13,473 queries. Table 2 provides statistics for this dataset, distinguishing queries with and without brands.
  • Dictionary Data:
    • ManDic: 78,539 phrases.
  • WikiDic: Derived from 1.9 million Japanese and 12.7 million English Wikipedia articles.
  • QryDic: Derived from past e-commerce search queries.
  • Training Data for PMI: Queries from the past two years, tokenized with MeCab and UniDic.

Key Insights and Ideas

  • Synergy of Dictionary and BERT-CRF: The paper highlights that neither dictionary matching nor BERT-CRF alone is sufficient. Their combination, with dictionary matching providing a strong initial signal and BERT-CRF handling the remaining complexities, yields superior results.
  • Importance of Japanese-Specific Refinement: The work underscores the critical need for dedicated query refinement methods for Japanese, as English-centric approaches are not directly applicable due to linguistic differences and data scarcity.
  • Impact on Downstream Tasks: The most significant insight is the tangible improvement in downstream NLP tasks, specifically e-commerce NER, directly attributable to the refined query data. This validates the practical utility of the proposed method beyond just query canonicalization.
  • Targeted Representation Update: The strategy of updating BERT representations only for dictionary-matched terms is key to preventing the model from being negatively influenced by potentially incorrect or malformed segments.

Key Results

  • Query Refinement Performance: The proposed two-stage method achieved a chunk-level F1-score of 83.86% on the query refinement task, outperforming baselines like MeCab (32.58%), re-trained MeCab (46.30%), BiLSTM-CRF (79.66%), and BERT-CRF alone (80.36%). Dictionary matching alone achieved 82.41%.
  • Brand Extraction Performance: When used to preprocess queries for a brand extraction NER task, the refined queries led to a significant improvement, boosting the F1-score by 11.7 points compared to using MeCab for preprocessing (56.7% vs. 45.0%).
  • Qualitative Example: Table 5 shows a concrete example where the refinement method successfully extracts "カネテツデリカフーズ" from a query that MeCab tokenization failed to process correctly.

Limitations and Future Work

  • QryDic Errors: The authors note that the QryDic creation process can sometimes lead to incorrect entries due to high PMI scores for certain n-grams, suggesting the need for better association computation methods for tri- and four-grams.
  • BERT-CRF Splitting Errors: BERT-CRF can sometimes mistakenly split compound terms (e.g., alphanumeric + Japanese characters) into multiple words.
  • Future Work: The paper suggests exploring better association computation for n-grams in QryDic and potentially more advanced methods to handle complex term splitting scenarios. The authors also mention that the query refinement system has already been adopted by their e-commerce partner, indicating successful real-world deployment.

Summary updated

BibTeX
@inproceedings{nakayama2024search,
  author = {Yuki Nakayama and Ryutaro Tatsushima and Erick Mendieta and Koji Murakami and Keiji Shinzato},
  title = {{Search Query Refinement for Japanese Named Entity Recognition in E-commerce Domain}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-industry.39},
}
Figure