“Devils Are in the Details”: Annotating Specificity of Clinical Advice from Medical Literature

Yingya Li, Bei YuNAACL 2022

TL;DR

This paper introduces a novel annotation schema to systematically capture the "specificity" of clinical advice found in medical research literature. By breaking down advice into components like agents, qualifications, reasoning, and confidence, they pave the way for NLP tools that can automatically analyze and verify the quality and consistency of medical recommendations.

What Problem Does This Paper Address?

Medical research papers often contain clinical advice, but the vagueness or lack of detail in this advice can hinder its practical implementation by healthcare professionals. Prior studies have shown that more specific guidelines lead to better adoption rates. However, existing research on clinical advice often lacks a systematic way to quantify and analyze this specificity, especially within the vast and rapidly growing body of medical literature. This paper aims to bridge this gap by developing a detailed framework to dissect and understand the nuances of clinical advice.

Methods

The authors developed a fine-grained annotation schema by combining insights from research on clinical guidelines and NLP approaches to specificity. This schema partitions a clinical advice sentence into four key dimensions:

  • Agents: Who is expected to carry out the advice (e.g., health practitioners, organizations).
  • Substantial Qualifications or Elaborations: Concrete details about what, who, when, where, and how to implement the advice. This is further broken down into:
  • Intervention: Specifics of treatment, therapy, doses, etc.
  • Target: Who or what receives the intervention.
  • Goal: The illness or symptom to be treated.
  • Chain of Reasoning: The justification or explanation behind the advice.
  • Confidence: The level of certainty expressed by the authors.

They then used this schema to annotate a dataset of health advice sentences extracted from medical research papers.

Datasets

The study utilized an open-access dataset of 10,848 sentences from medical research papers, from which 2,748 sentences were identified as health advice. For schema development and validation, a subset of 200 advice sentences was used, with 100 identified as "clinical advice" and 98 as "non-clinical advice." The research papers included various study designs (RCTs, observational studies) to ensure the generalizability of the schema.

Key Insights and Ideas

  • Specificity is multifaceted: The paper highlights that "specificity" isn't a single attribute but a combination of distinct components (agents, qualifications, reasoning, confidence).
  • Clinical advice in research differs from guidelines: Unlike professionally-designed clinical guidelines, advice in research papers often omits "agents" and "targets," and reasoning is less frequently provided.
  • Study design correlates with advice specificity: Research papers from higher evidence-level study designs (like RCTs) tend to provide more clinical advice, suggesting a link between study rigor and the explicitness of recommendations.
  • The "devil is in the details": The paper's title aptly captures the essence – subtle differences in phrasing and detail can significantly impact the clarity and usability of clinical advice.

Key Results

The developed annotation schema demonstrated high inter-coder agreement, with an overall Cohen's Kappa of 0.88 for distinguishing clinical from non-clinical advice, and strong agreement across all specificity dimensions (ranging from 0.87 to 0.98). This indicates the schema is robust and reliable for annotation. The analysis of the annotated data revealed that "intervention," "confidence," and "goal" were the most frequently mentioned specificity aspects in clinical advice from research papers.

Limitations and Future Work

The authors acknowledge that "agent" and "target" are often omitted in medical literature advice, which is a limitation of the advice itself rather than the schema. Future work will focus on developing NLP models for automatic classification and extraction of these specificity aspects. Such tools could support downstream applications like summarizing clinical advice, detecting inconsistencies, and verifying recommendations against evidence, ultimately improving the quality and impact of medical research.

Summary updated

BibTeX
@inproceedings{li2022devils,
  author = {Yingya Li and Bei Yu},
  title = {{“Devils Are in the Details”: Annotating Specificity of Clinical Advice from Medical Literature}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure