Abstract
Clinical assertion detection—the task of determining the status of medical concepts in patient records—is critical for accurate clinical Natural Language Processing (NLP). Traditional rule-based and deep learning methods often struggle with linguistic nuance and imbalanced data. This paper presents a novel approach using Large Language Models (LLMs) combined with advanced reasoning strategies (Chain-of-Thought, Self-Consistency, Tree of Thought) and Parameter-Efficient Fine-Tuning (PEFT) via Low-Rank Adaptation (LoRA). Our results demonstrate that a fine-tuned LLaMA2-7B model significantly outperforms existing benchmarks, achieving a micro-averaged F1-score of 0.89 on the i2b2 dataset and 0.74 on a specialized Sleep dataset.
Introduction
In clinical NLP, identifying a medical entity is insufficient for clinical utility. One must also determine its assertion status: is the condition present, absent, possible, or related to a family member? This task, known as assertion detection, is essential for constructing accurate patient profiles from unstructured electronic health record (EHR) notes.
Traditional approaches, such as the rule-based NegEx system, rely on rigid patterns and often fail to capture complex linguistic dependencies. While earlier deep learning models (CNNs/LSTMs) improved performance, they require extensive labeled data and frequently underperform on "minority" classes—assertion types like "Possible" or "Family" that occur less frequently but carry high clinical significance.
Methodology
To overcome these limitations, researchers from the University of Pittsburgh moved away from rigid rules toward a generative framework utilizing LLMs (specifically LLaMA2-7B and GPT-3.5). The methodology integrates advanced prompting techniques with efficient fine-tuning.
Advanced Reasoning Strategies
Three primary in-context learning strategies were explored to enhance the model's logical processing:
- Chain-of-Thought (CoT): Prompts the model to generate intermediate reasoning steps before providing a final classification.
- Self-Consistency (SC): Executes multiple reasoning paths and employs a majority-voting mechanism to arrive at a robust conclusion.
- Tree of Thoughts (ToT): A heuristic-guided approach where the model explores multiple reasoning branches, evaluating the logical alignment of each path with the clinical evidence.
Parameter-Efficient Fine-Tuning
The study utilized Low-Rank Adaptation (LoRA) to fine-tune the LLaMA2-7B model. LoRA allows for the adaptation of large-scale models to specific domains by training only a small subset of additional parameters, significantly reducing computational overhead while maintaining high performance.
Experimental Setup
The models were evaluated on two distinct datasets to test both raw performance and generalizability:
- i2b2 2010 Assertion Dataset: The industry standard benchmark, containing annotations for six assertion types (Present, Absent, Possible, Hypothetical, Conditional, and Family).
- Sleep Dataset (UPMC): A local, private corpus from the University of Pittsburgh Medical Center focused on sleep disorders like obstructive sleep apnea (OSA). This dataset added a "Historical" category, providing a more complex test of the model's ability to handle specialized medical domains.
Key Results and Discussion
The proposed LLM-based approach established new benchmarks across both datasets:
- i2b2 Dataset: Achieved a micro-averaged F1-score of 0.89, an improvement of 0.11 over previous state-of-the-art methods.
- Sleep Dataset: Achieved an F1-score of 0.74, which was 0.31 higher than traditional methods, proving the model's strong generalizability.
- LoRA Efficiency: The fine-tuned LLaMA2-7B consistently outperformed the much larger ChatGPT 3.5 in specific categories like "Negated" (0.98 vs 0.69) and "Positive" (0.99 vs 0.72) on the i2b2 dataset.
By treating assertion detection as a generative task, the researchers improved the interpretability of the results. The model can explain why it chose a label, making the system more trustworthy for healthcare professionals.
Conclusion and Future Work
This research demonstrates that LLMs, when paired with structured reasoning and efficient fine-tuning, offer a powerful solution for clinical assertion detection. While the model still occasionally struggles with implicit assertions and hardware constraints limited the study to 7B parameter models, the results mark a substantial leap in machine understanding of medical context. Future work will involve expanding the dataset to include notes from multiple healthcare systems to ensure the model's robustness across different institutional writing styles.
Summary updated