Incorporating Figure Captions and Descriptive Text in MeSH Term Indexing

Xindi Wang, Robert E. MercerACL 2019

TL;DR

This paper tackles the challenge of automatically assigning Medical Subject Headings (MeSH) terms to biomedical articles, a crucial but labor-intensive task. They introduce a novel deep learning model, multichannel TextCNN, that leverages not only abstracts and titles but also figure/table captions and associated paragraphs, significantly improving indexing accuracy.

What Problem Does This Paper Address?

Manually assigning MeSH terms to biomedical literature is essential for organizing and retrieving information from databases like MEDLINE and PubMed. However, this process is time-consuming and costly, especially given the massive influx of new research daily. While automatic MeSH indexing systems exist, they often rely solely on abstracts and titles. This paper argues that crucial contextual information, particularly related to figures and tables, is often overlooked, leading to suboptimal indexing. The authors aim to bridge this gap by incorporating these underutilized text sources into a deep learning framework.

Methods

The core of this work is a novel multichannel TextCNN model. Unlike traditional approaches that treat text as a single input, this model uses two distinct input channels:

  • Channel 1: Processes word embeddings from the article's abstract and title.
  • Channel 2: Processes word embeddings from figure and table captions, along with paragraphs that explicitly reference these figures and tables.

This multichannel architecture allows the model to learn and combine features from these different text sources independently before integrating them. The model employs convolutional neural networks (CNNs) with varying filter sizes (3, 4, and 5) to capture local patterns in the text. These features are then pooled and fed into a fully connected bottleneck layer, followed by a sigmoid classifier that outputs probabilities for each of the ~29,000 MeSH terms. The model is trained using binary cross-entropy loss, optimizing for a one-versus-all prediction for each MeSH term.

Datasets

The authors introduce a new, comprehensive dataset for MeSH term indexing. They utilize two primary sources:

  • 2015 Subject Extraction Test Collection (SETC2015): A smaller dataset of 14,828 PMC full-text articles. From this, they create two subsets:
  • AT (S): Abstract and Title only.
  • Full (S): Abstract, Title, figure/table captions, and associated paragraphs.
  • PMC Full Text Collection: A much larger dataset of 257,590 PMC full-text documents. From this, they create two larger subsets:
  • AT (L): Abstract and Title only.
  • Full (L): Abstract, Title, figure/table captions, and associated paragraphs.

These datasets are publicly available, contributing a valuable resource to the research community.

Key Insights and Ideas

  • Figures and tables are rich sources of MeSH information: The most significant insight is that captions and descriptive text surrounding figures and tables contain crucial information for accurate MeSH indexing, often complementing or providing unique insights not found in abstracts and titles alone.
  • Multichannel architecture is key: Separating and processing different text modalities (abstract/title vs. figure/table related text) in distinct channels allows the model to better capture and integrate diverse features, leading to improved performance.
  • Extreme multi-label classification is challenging: The sheer number of MeSH terms (~29,000) and their highly skewed distribution (some terms appear in millions of documents, others in just a few) makes this an "extreme multi-label" problem, requiring specialized approaches.

Key Results

The multichannel TextCNN model consistently outperformed single-channel models across various metrics and datasets.

  • Performance Gains: Integrating figure/table captions and associated paragraphs led to significant improvements. For example, on the AT (S) dataset, the multichannel model achieved a p@1 score of 0.78220, compared to 0.76197 for the single-channel model. Similar gains were observed across other metrics like p@k and nDCG@k.
  • Large vs. Small Datasets: The improvement from incorporating extra text was more pronounced on the smaller SETC2015 dataset (Small datasets) compared to the larger PMC dataset (Large datasets). The authors hypothesize this could be due to the larger dataset already having a stronger baseline performance or inherent differences in the document attributes.
  • Hierarchical Evaluation: Hierarchical evaluation metrics (HP and HR) showed that the model not only predicted relevant terms but also terms that were in the correct hierarchical branch of the MeSH taxonomy, indicating a deeper understanding of the document's content.

Limitations and Future Work

  • Hyperparameter Tuning: While the authors fixed hyperparameters for their experiments, they suggest future work could involve exploring different optimizers, learning rates, and classifiers to further improve model performance.
  • Automatic Label Count: The current model requires manually setting the number of top-k predicted MeSH terms. Future work aims to implement a ranking system module that can automatically suggest the optimal number of labels for each document.
  • Human Annotation Assistance: The authors plan to develop tools to help human annotators locate important text segments within documents, thereby improving the efficiency of computer-assisted human MeSH indexing.

Summary updated

BibTeX
@inproceedings{wang2019incorporating,
  author = {Xindi Wang and Robert E. Mercer},
  title = {{Incorporating Figure Captions and Descriptive Text in MeSH Term Indexing}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure