A Domain Knowledge Enhanced Pre-Trained Language Model for Vertical Search: Case Study on Medicinal Products

Kesong Liu, Jianhui Jiang, Feifei LyuCOLING 2022

TL;DR

This paper introduces a novel pre-training approach for language models specifically designed for medicinal product vertical search. By combining ELECTRA's efficient pre-training with a new "Product Attribute Prediction" task that leverages structured product data, they significantly improve understanding of medicinal product queries.

What Problem Does This Paper Address?

Traditional pre-trained language models (PLMs) excel at understanding general text but often lack the specialized knowledge required for domain-specific search, such as in the medicinal product sector. For medicinal product search, users don't just search for drug names; they also use symptoms, diseases, and other attributes. Existing PLMs, trained on general corpora or even medical literature, struggle to connect these diverse search terms to the structured information found in product databases (like drug names, indications, and side effects). This gap hinders effective query understanding and relevance matching in vertical search engines.

Methods

The authors build upon ELECTRA's "Replaced Token Detection" (RTD) pre-training strategy. Their key innovations are:

  • Biomedical Entity Masking (EM): Instead of masking random tokens, they dynamically mask spans of biomedical entities (drug names, diseases, symptoms, etc.) to better capture contextual representations within the biomedical domain.
  • Product Attribute Prediction (PAP) Task: This novel pre-training task injects domain-specific knowledge by leveraging structured medicinal product databases. It involves encoding product titles, attribute names, and attribute values into representations. The model is trained to predict whether a given attribute value is associated with a product title and attribute name, pushing related concepts closer and unrelated ones further apart using a contrastive loss. This is inspired by knowledge graph embedding techniques like TransE but applied directly to textual product information.
  • Multi-task Pre-training: The RTD (with EM) and PAP tasks are jointly learned, allowing the model to benefit from both general language understanding and specific product knowledge.

Datasets

  • Pre-training Corpora:
    • General Domain: Wikipedia, News Articles.
  • Biomedical Domain: Package Insert of Drugs, Medical Encyclopedia, Biomedical Community QA.
  • Medicinal Product Dataset (for PAP): A custom dataset constructed from medicinal product databases, containing product titles, attribute names, and attribute values.
  • Evaluation Benchmarks:
    • ChineseBLUE: A benchmark for general Chinese biomedical NLP tasks (e.g., named entity recognition, question answering).
  • Medicinal Product Vertical Search Benchmark: A newly constructed benchmark comprising four tasks:
  • QTRel-easy & QTRel-hard: Query-title relevance tasks. "Hard" indicates queries where no terms directly appear in the title.
  • QIC: Query intention classification (22 types).
  • QNER: Named entity recognition in queries (28 entity types).

Key Insights and Ideas

  • Structured Data is Key for Vertical Search: The paper strongly advocates for leveraging structured product information (like attribute-value pairs) for domain-specific search, rather than relying solely on unstructured text.
  • PAP Task's Effectiveness: The Product Attribute Prediction task is shown to be highly effective in injecting crucial medicinal product knowledge, leading to significant performance gains.
  • Complementary Nature of Tasks: Combining general language understanding (via RTD with EM) and domain-specific knowledge injection (via PAP) creates a more powerful model for the target vertical search scenario.
  • QTRel-hard as a Strong Indicator: The performance degradation on the "hard" query-title relevance task when PAP is removed suggests that PAP is particularly crucial for understanding nuanced relationships where direct keyword overlap is minimal.

Key Results

  • The proposed model, ELECTRA+EM+PAP, significantly outperforms baselines like ELECTRA-Base and ELECTRA+EM on all four medicinal product vertical search tasks (QTRel-easy, QTRel-hard, QIC, QNER).
  • For example, on QTRel-hard, ELECTRA+EM+PAP achieved an F1 score of 84.88/85.86, compared to ELECTRA-Base's 81.90/82.80.
  • The model also showed comparable or improved performance on general Chinese biomedical NLP tasks (ChineseBLUE) compared to other state-of-the-art models.
  • Qualitative analysis (Table 6) shows ELECTRA+EM+PAP assigning higher relevance scores to correct query-title pairs, demonstrating better understanding of semantic relevance.

Limitations and Future Work

  • Computational Resources: The authors mention that limited computational resources prevented them from exploring larger model sizes, suggesting that scaling up could further unleash the potential of the PAP task.
  • Alternative Knowledge Integration: They note that other self-supervised learning tasks could be combined with PAP.
  • Graph-based Approaches: The paper suggests that comparing their structured-text-based PAP approach with graph embedding methods for product knowledge integration could be an interesting future research direction.
  • Generalizability: While the PAP task is presented as applicable to other product verticals, the current study focuses specifically on medicinal products. Further research could explore its application to other domains (e.g., fashion, electronics).

Summary updated

BibTeX
@inproceedings{liu2022domain,
  author = {Kesong Liu and Jianhui Jiang and Feifei Lyu},
  title = {{A Domain Knowledge Enhanced Pre-Trained Language Model for Vertical Search: Case Study on Medicinal Products}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2022},
}
Figure