Can Character Embeddings Improve Cause-of-Death Classification for Verbal Autopsy Narratives?

Zhaodong Yan, Serena Jeblee, Graeme HirstACL 2019

TL;DR

This paper explores using character-level text representations, alongside traditional word embeddings, to improve the accuracy of classifying causes of death from verbal autopsy narratives. They found that character embeddings significantly boost performance on smaller datasets, suggesting their robustness to the noisy and error-prone nature of these narratives.

What Problem Does This Paper Address?

Globally, a significant portion of deaths, particularly in low- and middle-income countries, occur without a formally determined cause. Verbal Autopsy (VA) surveys are a crucial method for gathering information about these deaths, typically involving demographic data, multiple-choice questions, and free-text narratives describing the events leading up to death. While automated systems exist for VA coding, they often underutilize the rich information contained in the narrative text. This is problematic because VA narratives, despite being collected by lay interviewers and often containing spelling errors, inconsistent grammar, and local terms, can provide more detailed insights than structured data alone. Existing automated methods have either focused on structured data or used word-level text analysis, potentially missing nuances due to misspellings and variations common in VA narratives. This paper aims to bridge this gap by investigating whether incorporating character-level information can enhance cause-of-death classification from these challenging narratives.

Methods

The authors propose and evaluate several models for cause-of-death classification using verbal autopsy narratives. Their core innovation lies in incorporating character embeddings.

  • Word-based Model: This standard approach uses word embeddings (trained with word2vec CBOW) to represent each word in the narrative. A Convolutional Neural Network (CNN) with max-pooling and dropout is then used for classification.
  • Character-based Model: This model represents each character in the text as a vector. A CNN is also used here, but it operates on character sequences. A key advantage highlighted is its ability to handle out-of-vocabulary words and misspellings, which are prevalent in VA data.
  • Combined Models: Two methods for combining word and character information are explored:
  • Embedding Concatenation: Word embeddings are concatenated with character embeddings for each word. To handle variable word lengths, only the first 7 characters are considered.
  • Model Combination: Feature vectors from separate word-based and character-based CNNs (excluding their final layers) are concatenated before being fed into a final classification layer.

Datasets

The primary dataset used is from the Million Death Study (MDS), a program collecting VAs in India. It comprises:

  • 12,045 adult records
  • 1,851 child records
  • 572 neonatal records

These records are classified into broad cause-of-death categories (18 for adults, 9 for children, and 5 for neonates). The paper also creates a smaller subset of the adult data ("Adult small"), representing 10% of the original adult dataset, to specifically test the hypothesis that character embeddings are more beneficial for smaller datasets.

Key Insights and Ideas

  • Character Embeddings for Noisy Text: The core idea is that character-level representations are inherently more robust to the spelling errors, grammatical inconsistencies, and variations found in real-world, non-expert-generated text like VA narratives. Unlike word embeddings, which struggle with unknown or misspelled words, character embeddings can still derive meaning from the constituent characters.
  • Small Datasets Benefit Most: The most significant finding is that character information provides a substantial improvement in classification performance, particularly when dealing with smaller datasets. This suggests that the noise and variability in VA narratives are more detrimental to word-based models when data is scarce.
  • Embedding Concatenation Outperforms: Among the combined models, the simple concatenation of word and character embeddings (with a fixed character window) emerged as the best-performing approach across most metrics and datasets, demonstrating the value of directly fusing these representations.

Key Results

  • Improved Performance on Small Datasets: For the "Adult small" dataset, the embedding concatenation model achieved significantly better precision, recall, and F1 scores compared to the word-only model. For instance, the F1 score for embedding concatenation was 0.660, while the word embedding model was only 0.456.
  • Robustness to Data Size: While the word-based model's performance dropped sharply on the smaller adult dataset, the character-based and combined models showed a smaller decrease, indicating their greater robustness to data scarcity.
  • Mixed Performance Across Categories: While the embedding concatenation model generally performed well, the word-based model sometimes outperformed it on specific categories, particularly for the child dataset and certain adult categories with high physician agreement. This suggests that for very well-defined or frequent causes, word-level information might suffice.
  • Handling Ambiguous Categories: The combined model showed particular strength in classifying more ambiguous categories like "Ill-defined" and "Unspecified infections," which are often challenging due to their broad nature.

Limitations and Future Work

  • Dataset Size: The authors acknowledge that the paucity of VA data is a major obstacle. While their character-based approach shows promise for smaller datasets, larger datasets would allow for more robust training, potentially enabling the use of more complex models like ELMo.
  • Language Expansion: The current work is limited to English narratives. Future work includes expanding models to other languages like Portuguese and Hindi, where larger VA datasets exist.
  • Structured Data Integration: The authors plan to investigate incorporating structured data (demographics, multiple-choice answers) alongside the narrative text to further improve performance.
  • Advanced Language Models: Exploring language models with character information, such as ELMo, is a future direction, though it would require out-of-domain data for training due to the limited size of the VA dataset.

Summary updated

BibTeX
@inproceedings{yan2019can,
  author = {Zhaodong Yan and Serena Jeblee and Graeme Hirst},
  title = {{Can Character Embeddings Improve Cause-of-Death Classification for Verbal Autopsy Narratives?}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure