Role of Local Context in Automatic Deidentification of Ungrammatical, Fragmented Text

Tawanda Sibanda, Ozlem UzunerNAACL 2006

TL;DR

This paper introduces a novel approach to automatically de-identify medical discharge summaries by focusing on the local context of words, outperforming existing methods that rely on dictionaries or global context. This is crucial for making sensitive patient data available for research while preserving privacy, especially given the ungrammatical and fragmented nature of these medical texts.

What Problem Does This Paper Address?

De-identifying clinical records is essential for sharing them with researchers. However, existing methods often struggle with the unique characteristics of medical discharge summaries. These documents are frequently ungrammatical, fragmented, and contain ad hoc language, misspellings, and jargon. Crucially, they often include personal health information (PHI) that can be ambiguous (e.g., "Huntington" as a disease vs. a person's name) or not present in standard dictionaries (e.g., misspelled or foreign names). Traditional approaches, heavily reliant on dictionaries and heuristic rules, fail to effectively remove this challenging PHI, posing a risk to patient privacy.

Methods

The authors propose a de-identification system based on Support Vector Machines (SVMs). Unlike previous methods, their SVM heavily relies on a statistical representation of local context. This means the system analyzes the words immediately surrounding a target word (the word being classified) to determine if it's PHI. They engineered a set of 26 features, including:

  • The target word itself and its lemmatized form.
  • The word before and after the target word, and their lemmatized forms.
  • Bigrams (pairs of words) before and after the target word.
  • Part-of-speech tags for the target word and its neighbors.
  • Capitalization of the target word.
  • Length of the target word.
  • Presence of the target word and its neighbors in various dictionaries (names, locations, hospitals, months).
  • The section heading of the document where the word appears.

This approach is designed to be independent of external dictionaries and hand-tailored heuristics, making it more robust to variations and errors in the text.

Datasets

The paper utilized several corpora for training and evaluation:

  • Previously de-identified discharge summaries: These were re-identified with generic placeholders [REMOVED] and then used to generate new corpora.
  • Re-identified corpora:
    • One corpus had [REMOVED] tokens replaced with appropriate, fake PHI from dictionaries.
  • Another corpus had [REMOVED] tokens and other text replaced with lexical items ambiguous between PHI and non-PHI.
  • A third corpus had [REMOVED] tokens corresponding to names replaced with appropriately formatted entries that could not be found in dictionaries.
  • Authentic discharge summaries: These were used in the final stages of the project and contained real PHI.

Key Insights and Ideas

  • Local context reigns supreme: The study strongly suggests that for fragmented and ungrammatical text like medical discharge summaries, the immediate surrounding words (local context) are far more informative for de-identification than global context or dictionaries.
  • Dictionaries are insufficient: PHI often includes misspelled or foreign names, or words that are ambiguous between PHI and non-PHI. Dictionaries, by their nature, cannot capture this variability or ambiguity, making them unreliable for comprehensive de-identification.
  • Global context struggles with fragmentation: Systems relying on global context (e.g., relationships between entities across a sentence or document) are less effective when sentences are short, fragmented, and often contain only one entity, which is common in discharge summaries.
  • SVMs excel with local context: The SVM approach, by focusing on a rich statistical representation of local context, demonstrates superior performance in identifying PHI under challenging conditions.

Key Results

The SVM-based system consistently outperformed other approaches across various evaluations:

  • On re-identified corpora: The SVM achieved an F-measure of 97.2% for PHI, significantly outperforming IdentiFinder (67.4%) and a heuristic+dictionary approach.
  • On authentic discharge summaries: The SVM achieved an F-measure of 92.0% for PHI, again outperforming IdentiFinder and the heuristic+dictionary approach.
  • When dealing with ambiguous words: The SVM achieved 92.5% F-measure on ambiguous data, highlighting its ability to resolve ambiguity using context.
  • When PHI is not in dictionaries: The SVM achieved 95.0% F-measure on PHI not found in dictionaries, demonstrating its robustness to unseen or misspelled names.

Limitations and Future Work

  • Feature Importance: While the SVM performed well, the paper notes that 12 of its 26 features were dictionary-related. Future work could explore further enriching the feature set to automatically capture more varied local context.
  • Global Context's Limited Role: The authors acknowledge that global context, while beneficial in other domains, contributed little to de-identification in this specific corpus due to its fragmented nature. Future research might explore how to adapt global context models for such challenging text.
  • Simplification of SNoW and IdentiFinder: The paper mentions that IdentiFinder was tested on a pre-trained news corpus, potentially putting it at a disadvantage. Similarly, SNoW was evaluated only on entities it was designed to recognize. These factors might influence direct comparisons.
  • Future Directions: The authors suggest that the simplicity and effectiveness of their local context representation are promising for many tasks involving ungrammatical and fragmented text where global context is unreliable. This opens avenues for applying similar techniques to other domains with similar text characteristics.

Summary updated

BibTeX
@inproceedings{sibanda2006role,
  author = {Tawanda Sibanda and Ozlem Uzuner},
  title = {{Role of Local Context in Automatic Deidentification of Ungrammatical, Fragmented Text}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2006},
}
Figure