Interpretable Probabilistic Latent Variable Models for Automatic Annotation of Clinical Text.

Alexander Kotov, Mehedi Hasan, April Idalski Carcone, Ming Dong, Sylvie Naar‐King, Kathryn Brogan-HartliebPubMed 2015

TL;DR

This paper introduces two novel probabilistic models, Latent Class Allocation (LCA) and Discriminative Labeled Latent Dirichlet Allocation (DL-LDA), for automatically annotating clinical text. These models improve upon existing methods by explicitly distinguishing between general and class-specific words, leading to more accurate and interpretable clinical text analysis, particularly for motivational interview transcripts.

What Problem Does This Paper Address?

Manually annotating clinical text, especially lengthy interview transcripts, is a time-consuming and resource-intensive process. This is crucial for understanding patient communication, identifying effective therapeutic strategies, and refining intervention models. Existing automated methods often struggle with the nuances of clinical language, which can be informal, context-dependent, and lack explicit cues (like facial expressions or tone). Furthermore, for clinical applications, these automated systems need to be interpretable, allowing practitioners to understand why a particular annotation was made. This paper addresses the need for accurate, interpretable, and efficient automatic annotation of clinical text, specifically focusing on motivational interview transcripts for adolescent obesity interventions.

Methods

The authors propose two novel probabilistic latent variable models:

  • Latent Class Allocation (LCA): This model learns class-specific multinomial distributions for words. It uses a latent variable to decide whether a word in a fragment is general or specific to a particular class. This allows it to directly model class-specific vocabularies.
  • Discriminative Labeled Latent Dirichlet Allocation (DL-LDA): An extension of Labeled LDA, DL-LDA also distinguishes between general and class-specific words. However, it further breaks down class-specific words into topics (clusters of semantically related words). This provides a more structured approach to modeling class-specific language.

Both models are designed to be interpretable, meaning their learned parameters (like word distributions) can be understood by clinicians. They are compared against standard baselines like Naïve Bayes and Labeled LDA.

Datasets

The primary dataset used in this work consists of 2966 manually annotated fragments of motivational interview transcripts from obese adolescents. These transcripts were annotated using the "Minority Youth Sequential Code for Observing Process Exchanges" (MY-SCOPE) coding manual, resulting in five distinct classes: negative commitment language (CL-), positive commitment language (CL+), negative change talk (CT-), positive change talk (CT+), and ambivalence (AMB).

The paper also describes experiments on aggregated classes:

  • Aggregating CL+ and CL- into a single "CL" class, and CT+ and CT- into a single "CT" class, resulting in three broader classes (CL, CT, AMB).
  • Aggregating positive and negative sub-classes into broader "+" and "-" sentiment categories, along with AMB.

Key Insights and Ideas

  • Discriminativity Matters: Explicitly modeling the difference between general words (common across many contexts) and class-specific words (highly indicative of a particular label) significantly improves annotation performance. This is a core idea behind both LCA and DL-LDA.
  • Topic Granularity: While DL-LDA's approach of breaking down class-specific language into topics can improve recall, it doesn't necessarily boost precision or F1 scores compared to LCA, suggesting that the optimal level of granularity depends on the task.
  • Stopwords Can Be Informative: Contrary to common practice, removing stopwords (like "the," "a," "is") sometimes degraded classification performance, indicating that these common words can carry important contextual or sentiment information in certain clinical language categories.
  • Interpretability is Key: The proposed models offer interpretable outputs (e.g., characteristic words for each class), which is crucial for adoption by clinical practitioners who need to trust and understand the automated annotations.

Key Results

  • Superior Performance: Both LCA and DL-LDA consistently outperformed the baselines (Naïve Bayes and Labeled LDA) across various performance metrics (Recall, Precision, F1 score) in distinguishing the five original language categories.
  • DL-LDA leads in Recall: DL-LDA achieved the highest Recall across most experiments, suggesting it's good at identifying relevant fragments.
  • LCA excels in Precision and F1: LCA often achieved the best Precision and F1 scores, indicating a good balance between identifying relevant fragments and avoiding false positives.
  • Preprocessing Effects: The optimal preprocessing strategy varied by model. DL-LDA performed best without preprocessing, while Naïve Bayes, LCA, and L-LDA benefited most from stemming.
  • Aggregated Classes: In experiments with aggregated classes, LCA and DL-LDA continued to show strong performance, with DL-LDA particularly excelling in detecting speaker attitude (sentiment modality). For distinguishing positive/negative sentiment within CL and CT, DL-LDA achieved an F1 score of 0.633, while LCA achieved 0.656.

Limitations and Future Work

  • Parameter Tuning: The performance of DL-LDA is sensitive to the number of per-class topics, which needs to be optimized. The paper explores this optimization, finding small numbers of topics (2-3 or 5) often yield the best results.
  • Scalability: While the paper mentions that inference algorithms are adaptable to distributed environments, the current experimental setup is not explicitly described as distributed.
  • Generalizability: While the methods are presented as applicable to any clinical text, the experiments are specifically conducted on motivational interview transcripts. Future work could explore their effectiveness on other types of clinical data.
  • Interpretability Nuances: While interpretable, the paper could further explore how clinicians can best leverage the model outputs for practical insights.

Summary updated

Abstract

We propose Latent Class Allocation (LCA) and Discriminative Labeled Latent Dirichlet Allocation (DL-LDA), two novel interpretable probabilistic latent variable models for automatic annotation of clinical text. Both models separate the terms that are highly characteristic of textual fragments annotated with a given set of labels from other non-discriminative terms, but rely on generative processes with different structure of latent variables. LCA directly learns class-specific multinomials, while DL-LDA breaks them down into topics (clusters of semantically related words). Extensive experimental evaluation indicates that the proposed models outperform Naïve Bayes, a standard probabilistic classifier, and Labeled LDA, a state-of-the-art topic model for labeled corpora, on the task of automatic annotation of transcripts of motivational interviews, while the output of the proposed models can be easily interpreted by clinical practitioners.
BibTeX
@inproceedings{kotov2015interpretable,
  author = {Alexander Kotov and Mehedi Hasan and April Idalski Carcone and Ming Dong and Sylvie Naar‐King and Kathryn Brogan-Hartlieb},
  title = {{Interpretable Probabilistic Latent Variable Models for Automatic Annotation of Clinical Text.}},
  booktitle = {PubMed},
  year = {2015},
}
Figure