TL;DR
This paper tackles the challenge of automatically extracting appointment details (reason and time) from doctor-patient conversations, aiming to reduce physician documentation burden. They introduce a novel sequence tagging task and explore weak supervision techniques, achieving a strong F1 score of 79.32 with a domain-specific BERT model.
What Problem Does This Paper Address?
Physicians face a significant documentation burden from Electronic Health Records (EHRs), leading to burnout and reduced patient face-time. While machine learning can help extract information from medical conversations, automatically identifying appointment details like the reason and time has been an underexplored area. Furthermore, manual annotation of medical conversations is expensive and requires domain expertise. This paper addresses the need for an automated solution to extract appointment spans, proposing a method that leverages weak supervision to overcome annotation limitations.
Methods
The authors frame appointment span extraction as a sequence tagging problem. They explore three weak supervision approaches:
- Inaccurate Supervision: Utilizes off-the-shelf tools like UMLS and spaCy to automatically generate noisy labels for reason and time spans.
- Incomplete Supervision: Leverages a small subset of manually annotated data.
- Hybrid Supervision: Combines inaccurate and incomplete supervision sequentially, first training with inaccurate labels and then fine-tuning with incomplete ones to mitigate catastrophic forgetting.
They evaluate both generic and domain-specific ELMo and BERT embeddings with sequence tagging models. The best performing model is a domain-specific BERT variant using the hybrid supervision approach.
Datasets
The paper introduces a newly annotated dataset derived from 23,000 fully-consented and de-identified doctor-patient conversations. This corpus includes 43,000 utterance windows discussing appointments, with 3,200 utterance windows from 5,000 conversations specifically annotated for appointment reason and time spans. They also obtained annotations for appointment duration and frequency but excluded them due to infrequency.
Key Insights and Ideas
- Weak Supervision is Key: The paper demonstrates that weak supervision, particularly a hybrid approach combining noisy and limited clean data, can significantly improve performance in a data-scarce domain like medical conversation annotation.
- Domain-Specific Embeddings Matter: Domain-specific embeddings (BioELMo and BioBERT) show a notable performance improvement over generic embeddings, especially when combined with effective supervision strategies.
- Error Analysis Reveals Nuances: The error analysis highlights that "complete false positives" are a major source of errors, particularly for time spans. This is often due to multiple time phrases being present in an utterance, making it difficult for the model to identify the confirmed appointment time.
Key Results
The best performing model, a domain-specific BERT variant using hybrid weak supervision, achieved an F1 score of 79.32.
- Hybrid supervision consistently improved performance across all model variants compared to inaccurate or incomplete supervision alone.
- Domain-specific pre-training (BioELMo, BioBERT) provided a performance boost, especially for ELMo.
- BERT variants generally outperformed ELMo variants.
- For the best model (BioBERT-hybrid), the F1 score for reason spans was 82.36, while for time spans it was 69.01, indicating a greater challenge in extracting accurate time spans.
Limitations and Future Work
- Annotation Inconsistencies: The authors acknowledge that inconsistencies in manual annotations contribute to errors, particularly in identifying overlapping spans.
- Difficulty with Multiple Mentions: The model struggles to select the confirmed appointment reason or time when multiple mentions are present in an utterance, suggesting a need to model relations between mentions.
- Ambiguity in Spoken Language: The conversational and oral nature of the data introduces ambiguity that complicates span extraction.
- Future Work: The authors plan to explore data augmentation techniques and joint entity and relation extraction to improve performance on difficult examples. They also aim to investigate the generalization of their work to automatic transcripts, where transcription errors might pose additional challenges.
Summary updated