Assertion Detection Large Language Model In-context Learning LoRA Fine-tuning

Yuelyu Ji, Zeshui Yu, Yanshan WangarXiv 2024 · DOI · arXiv

Abstract

Clinical assertion detection—the task of determining the status of medical concepts in patient records—is critical for accurate clinical Natural Language Processing (NLP). Traditional rule-based and deep learning methods often struggle with linguistic nuance and imbalanced data. This paper presents a novel approach using Large Language Models (LLMs) combined with advanced reasoning strategies (Chain-of-Thought, Self-Consistency, Tree of Thought) and Parameter-Efficient Fine-Tuning (PEFT) via Low-Rank Adaptation (LoRA). Our results demonstrate that a fine-tuned LLaMA2-7B model significantly outperforms existing benchmarks, achieving a micro-averaged F1-score of 0.89 on the i2b2 dataset and 0.74 on a specialized Sleep dataset.

Introduction

In clinical NLP, identifying a medical entity is insufficient for clinical utility. One must also determine its assertion status: is the condition present, absent, possible, or related to a family member? This task, known as assertion detection, is essential for constructing accurate patient profiles from unstructured electronic health record (EHR) notes.

Examples of clinical assertion categories and their corresponding textual cues in medical notes.

Traditional approaches, such as the rule-based NegEx system, rely on rigid patterns and often fail to capture complex linguistic dependencies. While earlier deep learning models (CNNs/LSTMs) improved performance, they require extensive labeled data and frequently underperform on "minority" classes—assertion types like "Possible" or "Family" that occur less frequently but carry high clinical significance.

Methodology

To overcome these limitations, researchers from the University of Pittsburgh moved away from rigid rules toward a generative framework utilizing LLMs (specifically LLaMA2-7B and GPT-3.5). The methodology integrates advanced prompting techniques with efficient fine-tuning.

Advanced Reasoning Strategies

Three primary in-context learning strategies were explored to enhance the model's logical processing:

  1. Chain-of-Thought (CoT): Prompts the model to generate intermediate reasoning steps before providing a final classification.
  2. Self-Consistency (SC): Executes multiple reasoning paths and employs a majority-voting mechanism to arrive at a robust conclusion.
  3. Tree of Thoughts (ToT): A heuristic-guided approach where the model explores multiple reasoning branches, evaluating the logical alignment of each path with the clinical evidence.

Parameter-Efficient Fine-Tuning

The study utilized Low-Rank Adaptation (LoRA) to fine-tune the LLaMA2-7B model. LoRA allows for the adaptation of large-scale models to specific domains by training only a small subset of additional parameters, significantly reducing computational overhead while maintaining high performance.

The comprehensive assertion detection pipeline, illustrating the input processing through Chain-of-Thought, Self-Consistency, Tree of Thought, and the final LoRA fine-tuned model.

Experimental Setup

The models were evaluated on two distinct datasets to test both raw performance and generalizability:

  • i2b2 2010 Assertion Dataset: The industry standard benchmark, containing annotations for six assertion types (Present, Absent, Possible, Hypothetical, Conditional, and Family).
  • Sleep Dataset (UPMC): A local, private corpus from the University of Pittsburgh Medical Center focused on sleep disorders like obstructive sleep apnea (OSA). This dataset added a "Historical" category, providing a more complex test of the model's ability to handle specialized medical domains.

Key Results and Discussion

The proposed LLM-based approach established new benchmarks across both datasets:

  • i2b2 Dataset: Achieved a micro-averaged F1-score of 0.89, an improvement of 0.11 over previous state-of-the-art methods.
  • Sleep Dataset: Achieved an F1-score of 0.74, which was 0.31 higher than traditional methods, proving the model's strong generalizability.
  • LoRA Efficiency: The fine-tuned LLaMA2-7B consistently outperformed the much larger ChatGPT 3.5 in specific categories like "Negated" (0.98 vs 0.69) and "Positive" (0.99 vs 0.72) on the i2b2 dataset.

By treating assertion detection as a generative task, the researchers improved the interpretability of the results. The model can explain why it chose a label, making the system more trustworthy for healthcare professionals.

Conclusion and Future Work

This research demonstrates that LLMs, when paired with structured reasoning and efficient fine-tuning, offer a powerful solution for clinical assertion detection. While the model still occasionally struggles with implicit assertions and hardware constraints limited the study to 7B parameter models, the results mark a substantial leap in machine understanding of medical context. Future work will involve expanding the dataset to include notes from multiple healthcare systems to ensure the model's robustness across different institutional writing styles.

Summary updated

Abstract

In this study, we aim to address the task of assertion detection when extracting medical concepts from clinical notes, a key process in clinical natural language processing (NLP). Assertion detection in clinical NLP usually involves identifying assertion types for medical concepts in the clinical text, namely certainty (whether the medical concept is positive, negated, possible, or hypothetical), temporality (whether the medical concept is for present or the past history), and experiencer (whether the medical concept is described for the patient or a family member). These assertion types are essential for healthcare professionals to quickly and clearly understand the context of medical conditions from unstructured clinical texts, directly influencing the quality and outcomes of patient care. Although widely used, traditional methods, particularly rule-based NLP systems and machine learning or deep learning models, demand intensive manual efforts to create patterns and tend to overlook less common assertion types, leading to an incomplete understanding of the context. To address this challenge, our research introduces a novel methodology that utilizes Large Language Models (LLMs) pre-trained on a vast array of medical data for assertion detection. We enhanced the current method with advanced reasoning techniques, including Tree of Thought (ToT), Chain of Thought (CoT), and Self-Consistency (SC), and refine it further with Low-Rank Adaptation (LoRA) fine-tuning. We first evaluated the model on the i2b2 2010 assertion dataset. Our method achieved a micro-averaged F-1 of 0.89, with 0.11 improvements over the previous works. To further assess the generalizability of our approach, we extended our evaluation to a local dataset that focused on sleep concept extraction. Our approach achieved an F-1 of 0.74, which is 0.31 higher than the previous method.
BibTeX
@misc{ji2024assertion,
  author = {Yuelyu Ji and Zeshui Yu and Yanshan Wang},
  title = {{Assertion Detection Large Language Model In-context Learning LoRA Fine-tuning}},
  year = {2024},
  eprint = {2401.17602},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2401.17602},
}
Figure