AttentionMeSH: Simple, Effective and Interpretable Automatic MeSH Indexer

Qiao Jin, Bhuwan Dhingra, William Cohen, Xinghua LuEMNLP 2018

TL;DR

This paper introduces AttentionMeSH, a novel deep learning model for automatically assigning Medical Subject Headings (MeSH) to biomedical articles. It achieves state-of-the-art performance while offering unprecedented word-level interpretability, allowing users to see which parts of the text support each assigned MeSH term.

What Problem Does This Paper Address?

The PubMed database contains millions of biomedical articles, and accurate indexing with MeSH terms is crucial for effective information retrieval. Manual MeSH assignment is a time-consuming and expensive process, costing approximately $9.40 per article. While automatic MeSH indexing models exist, they often lack interpretability, making it difficult for human indexers to trust and utilize their suggestions. Furthermore, the sheer number of MeSH terms (around 28,000) and the severe class imbalance (some terms are assigned to millions of articles, others to only a few hundred) present significant challenges for automatic systems. This paper aims to bridge this gap by developing a model that is both accurate and interpretable, assisting human indexers rather than replacing them entirely.

Methods

AttentionMeSH employs a deep learning architecture that combines a bidirectional gated recurrent unit (BiGRU) for context-aware word representations with an attention mechanism. Here's a breakdown of the key components:

  • BiGRU for Document Representation: The model first tokenizes the article's title, journal name, and abstract. These tokens are then converted into word embeddings, which are fed into a BiGRU to generate context-aware representations for each word.
  • Masking Mechanism: To handle the vast number of MeSH terms, a novel masking mechanism is introduced. Instead of considering all 28,000 MeSH terms for every article, the model first identifies a subset of candidate MeSH terms. This is done by finding the K nearest neighbors of the article in the training corpus (based on IDF-weighted word vectors) and selecting the most frequent MeSH terms among these neighbors. This significantly reduces the computational burden and focuses the model on more relevant terms.
  • Attention Mechanism: For each candidate MeSH term, an attention mechanism is applied. This mechanism calculates attention weights for each word in the document, indicating its relevance to that specific MeSH term. These weights are then used to create a MeSH-specific document representation by taking a weighted sum of the word vectors.
  • Classification: Finally, these MeSH-specific document representations are fed into nonlinear layers to classify each candidate MeSH term, outputting a probability of it being relevant to the article.
  • Interpretability: The attention weights themselves provide word-level interpretability. By visualizing these weights, users can see which words in the abstract contributed most to the assignment of a particular MeSH term.

Datasets

The primary dataset used is from the BioASQ challenge, which contains approximately 13.5 million manually annotated PubMed articles. For training, the authors selected 3 million articles from 2012 to 2017. They also used two test sets: the BioASQ test sets (provided weekly during the challenge) and a larger set of 100,000 latest articles for more precise evaluation.

Key Insights and Ideas

  • Interpretability is Crucial: The paper strongly argues that interpretability is not just a nice-to-have but a necessity for automatic indexing systems, as human indexers need to understand why a term is suggested to effectively use the system.
  • Masking for Scalability: The novel masking mechanism is key to making the model scalable to the large number of MeSH terms, significantly improving efficiency without sacrificing accuracy.
  • Attention for Localized Evidence: The attention mechanism effectively links textual evidence to MeSH term predictions, providing a granular level of interpretability that was previously lacking.
  • Soft-Attention's Power: The use of soft attention allows for a nuanced weighting of word importance, leading to more accurate and interpretable document representations tailored to specific MeSH terms.

Key Results

  • Performance: The AttentionMeSH model achieved a Micro-F (MiF) score of approximately 0.684 on their test set, which is close to state-of-the-art performance. During the BioASQ Challenge Task6a, an on-construction version of the model ranked second by average MiF.
  • Interpretability: Human evaluations showed that AttentionMeSH retrieves about 90% of all expert-labeled relevant words for a given MeSH-article pair at 20 outputs, demonstrating high interpretability.
  • Masking Effectiveness: Experiments showed that the masking layer, by selecting relevant candidate MeSH terms, significantly improves performance and efficiency. For instance, using 1000 nearest neighbors and a mask size of 1024 achieved a micro-recall of 0.97, ensuring most true annotations were included as candidates.

Limitations and Future Work

  • Abstract-Centric Indexing: A significant limitation acknowledged is that the model, like most current automatic indexers, only uses abstracts, titles, and journal names. Human indexers, however, examine the full article bodies, meaning crucial textual evidence might be missed during training. This leads to an inherent upper bound on performance due to potential false negatives.
  • False Positives: The authors note that AttentionMeSH sometimes predicts many MeSH terms that are not annotated by NLM indexers, which can appear as "false positives." However, they argue these predictions are often semantically sensible and might highlight inconsistencies in manual indexing.
  • Future Work: The authors suggest that a more convincing evaluation would involve a user study with NLM indexers to measure annotation efficiency and accuracy with and without the help of AttentionMeSH. They also highlight the need for models that can leverage full article bodies for more comprehensive indexing.

Summary updated

BibTeX
@inproceedings{jin2018attentionmesh,
  author = {Qiao Jin and Bhuwan Dhingra and William Cohen and Xinghua Lu},
  title = {{AttentionMeSH: Simple, Effective and Interpretable Automatic MeSH Indexer}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure