UTA DLNLP at SemEval-2016 Task 12: Deep Learning Based Natural Language Processing System for Clinical Information Identification from Clinical Notes and Pathology Reports

Peng Li, Heng HuangSemEval @ NAACL 2016 · DOI

Abstract

This paper presents a deep learning-based system developed by researchers at the University of Texas at Arlington for SemEval-2016 Task 12. The system is designed to automatically extract clinical events and their associated attributes from unstructured clinical notes and pathology reports. By utilizing 1D Convolutional Neural Networks (CNNs) to learn feature representations directly from text, the approach reduces the reliance on manual feature engineering. Evaluation on the Clinical TempEval corpus shows that the system outperforms traditional baselines in identifying event spans and classifying attributes such as polarity, degree, and modality.

Introduction

Extracting structured information from Electronic Medical Records (EMRs) is a significant challenge in clinical informatics. Clinical text is often "noisy," containing numerous abbreviations, non-standard formatting, and domain-specific jargon. Traditionally, this task has required extensive "feature engineering," where experts manually create rules or use complex software suites like Apache cTAKES to identify linguistic patterns.

The authors propose a deep representation learning approach using temporal CNNs to identify clinical event spans and their properties. The goal is to determine if a model can learn to identify these features—such as whether a symptom is current or hypothetical—with minimal human intervention.

Example of clinical note annotation showing event mentions (e.g., bleeding, nausea) and their corresponding attributes like polarity and modality.

Methods

The system employs a three-stage pipeline to transform raw clinical text into structured data:

  1. Preprocessing & Tokenization: Due to the complexity of clinical text, a custom RegexpTokenizer from NLTK was used to handle character offsets precisely. Part-of-Speech (POS) tagging was performed using a "PerceptronTagger."
  2. Feature Representation: Each word is represented by concatenating three features: pre-trained 300-dimensional GloVe word embeddings, POS tags for grammatical context, and "word shape" features (e.g., mapping "April" to "Xxxxx"). Word shapes help the model recognize patterns in medical formatting even for out-of-vocabulary terms.
  3. Neural Architecture: A 1D Temporal CNN slides a filter over word windows to capture local context. After a max-pooling layer identifies the most significant features, a Multilayer Perceptron (MLP) with a softmax output layer predicts the event spans and their attributes.

Datasets

The study utilized the Clinical TempEval corpus (THYME data), which consists of 600 clinical notes and pathology reports from cancer patients at the Mayo Clinic. The data was split into a training set of 293 documents (38,872 events), a development set of 147 documents (20,973 events), and a testing set of 151 documents (18,989 events).

Results and Discussion

The system was evaluated on its ability to identify event spans and four key attributes: Modality, Degree, Polarity, and Type.

Performance metrics (Precision, Recall, F1) for event span identification and attribute classification across different system runs and baselines.

Key results include:

  • Event Span Identification: The system achieved an F1-score of 0.874, outperforming the "Memorize" baseline of 0.855.
  • Attribute Classification: The model performed consistently well, with F1-scores for Polarity (0.842) and Degree (0.870) leading the results.
  • DocTimeRel: For relating events to document time, the "RUN5" configuration achieved an F1 of 0.788, which was notably higher than the median report of 0.724.

The authors found that choosing the right tokenizer was critical for identifying exact character offsets. Furthermore, the inclusion of word shape features provided a lightweight method for the model to understand the structure of tokens like dosages or stages.

Conclusion

The UTA DLNLP system demonstrates that general-purpose deep learning architectures can compete with or exceed the performance of specialized clinical NLP tools without extensive manual tuning. While the system showed high precision, the authors noted lower recall, likely due to general-purpose word embeddings not fully covering specific medical terminology. Future work will focus on integrating domain-specific embeddings and hybrid approaches to improve the capture of nuanced clinical language.

Summary updated

BibTeX
@inproceedings{li2016uta,
  author = {Peng Li and Heng Huang},
  title = {{UTA DLNLP at SemEval-2016 Task 12: Deep Learning Based Natural Language Processing System for Clinical Information Identification from Clinical Notes and Pathology Reports}},
  booktitle = {SemEval @ NAACL},
  year = {2016},
  doi = {10.18653/v1/s16-1197},
}
Figure