Towards Extracting Medical Family History from Natural Language Interactions: A New Dataset and Baselines

Mahmoud Azab, Stephane Dadian, Vivi Nastase, Larry An, Rada MihalceaEMNLP 2019 · DOI

Introduction

Collecting a patient’s family medical history is a cornerstone of preventative medicine, particularly for identifying risks associated with hereditary diseases such as cancer. However, the current reliance on in-person genetic counseling creates a significant bottleneck due to time-intensive appointments and long waitlists. While automated systems like chatbots offer a potential solution, they face a fundamental trade-off: rigid, structured questionnaires are tedious for patients, whereas natural, narrative responses are notoriously difficult for computational systems to parse.

This research addresses this gap by introducing the first large-scale, publicly available dataset of natural language interactions specifically annotated for medical family history, alongside a robust baseline NLP pipeline for relation extraction.

The Family History Dataset

The study introduces a new, expert-annotated dataset comprising 228 family histories. These histories were gathered from two primary sources: 156 collected via Amazon Mechanical Turk (AMT) and 72 derived from actual genetic counseling session (GCS) test cases.

The dataset is characterized by its rich annotations, which go beyond simple entity recognition to include attributes such as age at diagnosis, cause of death, and complex relationships extending to the third degree (e.g., great-grandparents and cousins). In total, the corpus contains 4,304 sentences and 20,774 annotated relations.

Table 1: Distribution of annotation types across the AMT and GCS datasets, highlighting the depth of the medical information captured.

Methodology: A Two-Stage Pipeline

To extract structured data from narrative text, the authors developed a two-stage pipeline focused on Relation Extraction.

1. Entity Identification

The first stage employs a Conditional Random Field (CRF) classifier using IOB (Inside, Outside, Beginning) tagging. This system identifies key entities within a sentence, such as family members (FT), names, ages, genders, and specific illnesses.

2. Relation Classification

Once entities are identified, the system must determine the semantic links between them—for instance, associating a specific age of diagnosis with the correct relative. This is achieved using a bi-directional LSTM (Bi-LSTM) classifier. For every pair of entities identified in a sentence, the model predicts the existence of a valid medical relation based on GloVe word embeddings, part-of-speech tags, and the relative positions of the entities.

Figure 1: The two-stage architecture for relation extraction, moving from raw text to entity tagging and finally to deep neural network-based relation classification.

Key Insights and Results

The baseline system demonstrated strong performance, achieving an overall F-score of 0.87 on complex sentences (those containing multiple relatives). When the system was evaluated solely on its ability to link pre-identified entities, the F-score rose to 0.933.

Several key insights emerged from the analysis:

  • Narrative Elicitation: Choosing to elicit narrative responses rather than structured ones proved more patient-friendly and resulted in more comprehensive data collection.
  • Data Density: AMT-sourced data was significantly denser than clinical GCS data, averaging 14.2 relations per complex sentence compared to 6.1.
  • Cross-Domain Robustness: Models trained on crowdsourced data performed effectively when tested on clinical data, suggesting that crowdsourcing is a viable method for developing real-world medical NLP applications.

Conclusion

This work provides a foundational resource for the medical NLP community. By releasing a high-quality "gold standard" dataset (with an inter-annotator agreement Cohen’s Kappa of 0.8908), the authors enable future research into automated family history collection. Future work will focus on expanding the scope beyond cancer history and developing active "Dialog Agents" capable of real-time, intelligent follow-up questions.

The dataset and baselines are available at http://lit.eecs.umich.edu/downloads.html.

Summary updated

BibTeX
@inproceedings{azab2019towards,
  author = {Mahmoud Azab and Stephane Dadian and Vivi Nastase and Larry An and Rada Mihalcea},
  title = {{Towards Extracting Medical Family History from Natural Language Interactions: A New Dataset and Baselines}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  doi = {10.18653/v1/d19-1122},
}
Figure