Syntactic Patterns Improve Information Extraction for Medical Search

Roma Patel, Yinfei Yang, Iain Marshall, Ani Nenkova, Byron Wallace

TL;DR

By leveraging minimally supervised syntactic patterns (like "patients with" or "treated with"), this paper demonstrates a significant performance boost in extracting key medical information (PICO elements) from unstructured clinical abstracts. Integrating these patterns into state-of-the-art sequence tagging models—either as explicit features or as distinct semantic embeddings—captures crucial contextual information that standard word-level representations completely miss.

What Problem Does This Paper Address?

When medical professionals search clinical literature, they typically look for specific components defined by the PICO framework: Participants (e.g., disease, age), Interventions/Comparators (e.g., drug dosage), and Outcomes (e.g., survival rate). Unfortunately, the vast majority of medical abstracts are unstructured, making it difficult for standard search engines to pinpoint these details accurately.

While automated sequence labeling models (like CRFs and LSTM-CRFs) can help extract these text spans, they often struggle with ambiguity. For instance, the word "children" might describe the target patient population in one sentence, or simply be a passing mention in another. The authors address this challenge by capturing the syntactic context surrounding these entities to improve extraction accuracy.

Methods

The authors propose a pipeline that extracts syntactic patterns using AutoSlog-TS, a minimally supervised system. By analyzing a subset of PubMed abstracts that possess explicit structural headings (e.g., "Participants:", "Interventions:"), AutoSlog-TS identifies characteristic $n$-gram patterns (mostly bigrams) that frequently appear in these specific sections.

They explore two main ways to feed these patterns into supervised sequence tagging models:

  1. Linear CRF Enrichment: Adding binary features to a standard Linear-chain CRF indicating whether a token or its immediate neighbors match an extracted syntactic pattern.
  2. LSTM-CRF Integration: Incorporating patterns into a deep learning architecture via three distinct strategies:
  3. Before CRF: Appending pattern indicator features directly to the output of the BiLSTM layer.
  4. Before BiLSTM: Concatenating pattern indicators to the word and character embeddings at the input layer.
  5. Pattern Embeddings: Treating multi-token patterns as single units (collocations) and training joint 200-dimensional Word2Vec (Skip-Gram) embeddings so that patterns and unigrams share the same semantic space.

Datasets

  • Labeled Data: A corpus of 4,741 medical abstracts with crowd-sourced annotations for P, I, and O sequences used for training, and a high-quality test set of 191 abstracts annotated by medical professionals.
  • Pattern Extraction Data: 50,000 text segments for each P, I, and O category extracted automatically from structured PubMed abstracts.
  • Embedding Pre-training Data: 6 million PubMed abstracts (~1.4 billion words) filtered specifically for Human Randomized Controlled Trials (RCTs) to train the custom Word2Vec pattern embeddings.

Key Insights and Ideas

  • The Power of Function Words: The vast majority (82.86%) of the highly predictive patterns consist of a content word paired with a function/stopword (e.g., "patients with" vs. "patients were"). The function word acts as a syntactic anchor that completely changes the meaning and role of the content word.
  • Semantic Space Realignment: When multi-token patterns are embedded as single units, they undergo a dramatic semantic shift. Instead of clustering near their literal constituent words, they cluster near topically relevant concepts. For example, the pattern <side effect> moves away from the individual words "side" or "effect" and clusters tightly with actual clinical symptoms like headache and fatigue.
Figure 1: Scatter of PCA-reduced embeddings clus- tered using K-means. <> brackets show the syntactic pattern n-grams given by Autoslog-TS that are embed- ding in the same space as unigrams.

Key Results

  • Massive Recall Boost for CRFs: Incorporating syntactic patterns into the baseline Linear CRF led to substantial improvements across the board. Most notably, the recall for Outcome (O) extraction skyrocketed from a dismal 10.47% to 44.07%.
  • Optimal Neural Strategy: For the LSTM-CRF architecture, injecting the pattern features before the BiLSTM layer yielded the best overall performance, achieving an F1-score of 69.97% for Participants and 49.74% for Interventions.
Performance comparison of various models with and without pattern features.
  • Syntax vs. Generic Bigrams: The authors verified that the boost was truly due to syntactic structure rather than just raw bigram frequency. The syntax-driven AutoSlog patterns significantly outperformed generic bigrams and standard word2vec collocations.

Limitations and Future Work

A minor limitation noted in the study is that the authors relied on standard, off-the-shelf hyperparameters for the LSTM models without task-specific optimization. Furthermore, the dataset used to train the pattern embeddings (~1.4 billion words) was considerably smaller than the massive multi-billion-word generic medical corpora typically used to train standard unigram embeddings. The authors suggest that scaling up the corpus for pattern embedding pre-training could unlock even greater performance gains.

Summary updated

Abstract

Medical professionals search the published literature by specifying the type of patients, the medical intervention(s) and the outcome measure(s) of interest. In this paper we demonstrate how features encoding syntactic patterns improve the performance of state-of-the-art sequence tagging models (both linear and neural) for information extraction of these medically relevant categories. We present an analysis of the type of patterns exploited, and the semantic space induced for these, i.e., the distributed representations learned for identified multi-token patterns. We show that these learned representations differ substantially from those of the constituent unigrams, suggesting that the patterns capture contextual information that is otherwise lost.
BibTeX
@inproceedings{patel2018syntactic,
  author = {Roma Patel and Yinfei Yang and Iain Marshall and Ani Nenkova and Byron Wallace},
  title = {{Syntactic Patterns Improve Information Extraction for Medical Search}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
  doi = {10.18653/v1/n18-2060},
}
Figure