KW-ATTN: Knowledge Infused Attention for Accurate and Interpretable Text Classification

Hyeju Jang, Seojin Bang, Wen Xiao, Giuseppe Carenini, Raymond T. Ng, Young Ji LeeDeeLIO @ NAACL 2021 · OpenReview

Abstract

Hyeju Jang, Seojin Bang, Wen Xiao, Giuseppe Carenini, Raymond Ng, Young ji Lee. Proceedings of Deep Learning Inside Out (DeeLIO): The 2nd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures.

2021.

Introduction

Deep learning has revolutionized text classification, yet it faces two persistent hurdles: high data requirements and the "black box" nature of neural predictions. In specialized fields like medicine, large labeled datasets are often unavailable, and model transparency is critical for clinical trust. To address these gaps, we present KW-ATTN (KnoWledge infused ATTentioN), a novel mechanism that integrates high-level concepts from external knowledge bases to improve both classification accuracy and model interpretability.

Methodology: The KW-ATTN Framework

The core contribution of this work is an attention mechanism that processes raw text alongside corresponding high-level concepts retrieved from external ontologies. Unlike standard attention that focuses solely on words, KW-ATTN utilizes a dual-stream architecture.

The KW-ATTN architecture features parallel word and concept encoders with a dual-layered attention mechanism.

As shown in the architecture diagram, the framework introduces a two-layered attention strategy:

  • Combined Attention (α): Determines the importance of a specific position in a sentence relative to the overall meaning.
  • Balancing Attention (p): This dynamic component allows the model to decide how much to trust a specific word versus its generalized concept. For example, the model might learn that the broader concept "athletics" is more predictive than the specific term "archery" when training data is sparse.

KW-ATTN is designed to be "plug-and-play," compatible with Recurrent Neural Networks (RNNs), Hierarchical Attention Networks (HAN), and transformer-based models like BERT.

Experimental Evaluation

The authors evaluated KW-ATTN on two distinct tasks using specialized knowledge bases:

  1. Patient Need Detection: A domain-specific dataset of 853 posts from cancer discussion forums, utilizing the Unified Medical Language System (UMLS).
  2. Topic Classification: A subset of 10,000 instances from Yahoo Answers, utilizing BabelNet for general-domain semantic knowledge.

Results and Key Insights

KW-ATTN consistently outperformed standard attention baselines and previous concept-integration methods. Key findings include:

  • Small Data Efficiency: The performance gap was most significant on limited data. On the Yahoo dataset with only 2,500 samples, KW-ATTN improved the F1 score by over 6% compared to standard attention.
  • BERT Enhancement: Even when added to pre-trained models like BERT, KW-ATTN provided a statistically significant boost, particularly in the medical domain where BERT’s general-purpose knowledge is less comprehensive.
  • Domain Specificity: On medical data, the specialized UMLS knowledge base yielded superior results compared to the general-purpose BabelNet, highlighting the importance of knowledge relevance.
  • Quantifiable Interpretability: Crowdsourced evaluations via Amazon Mechanical Turk revealed that users have significantly higher confidence in predictions when explanations include high-level concepts (e.g., "java as a programming language") rather than raw words alone.

Limitations and Future Work

The efficacy of KW-ATTN is dependent on the quality of the external knowledge base and the accuracy of automated concept-linking tools (e.g., MetaMap or Babelfy).

Examples of UMLS semantic types assigned to expressions, highlighting potential annotation noise (e.g., "Mom" as a Quantitative Concept).

As illustrated above, automated tools can introduce noise; for instance, MetaMap incorrectly tagged the word "Mom" as a "Quantitative Concept." Future research will explore utilizing the structural relationships between concepts within knowledge graphs, such as through Graph Neural Networks, to further refine the integration of external knowledge.

Summary updated

BibTeX
@inproceedings{jang2021kw,
  author = {Hyeju Jang and Seojin Bang and Wen Xiao and Giuseppe Carenini and Raymond T. Ng and Young Ji Lee},
  title = {{KW-ATTN: Knowledge Infused Attention for Accurate and Interpretable Text Classification}},
  booktitle = {DeeLIO @ NAACL},
  year = {2021},
  url = {https://openreview.net/forum?id=w3fkOFbLrp},
}
Figure