Subword Variation in Text Message Classification

Robert Munro, Christopher D. Manning

Abstract

Automated text classification in low-resource, morphologically rich languages faces significant hurdles due to brevity, complex word structures, and orthographic variation. This paper investigates subword modeling techniques—specifically morphological segmentation and phonological normalization—applied to medical SMS messages in Chichewa. Our results demonstrate that moving below the word level yields substantial accuracy improvements, achieving up to a 63.8% error reduction and significantly closing the performance gap between Chichewa and English baselines.

Introduction: The SMS Challenge in Rural Health

In many developing regions, text messaging (SMS) serves as a critical "leapfrog" technology for healthcare and emergency response. However, automating the processing of these messages is difficult. Unlike English, which has relatively simple morphology, Bantu languages like Chichewa are morphologically complex; a single word can contain multiple morphemes representing subjects, tenses, and objects. Furthermore, SMS users often employ phonetic spelling or regional variations, creating a sparse and noisy feature space for traditional word-based classifiers.

Methods: Modeling Below the Word Level

To address these challenges, the authors developed models that account for subword variation through two primary lenses:

1. Morphological Modeling

The study compared language-specific parsers based on formal grammar with language-independent unsupervised models. The latter utilized a Hierarchical Dirichlet Process (HDP) to discover morphemes without pre-defined dictionaries. Additionally, a "Morph-config" approach was introduced, treating sequences of discovered morphemes as features to capture structural information lost in simple bag-of-words models.

2. Phonological and Orthographic Normalization

To handle spelling variations, the researchers implemented heuristic rules for common Chichewa patterns and script-specific normalization. A conservative normalization strategy was employed to ensure that transformed words existed elsewhere in the corpus, preventing the introduction of "hallucinated" terms. They also explored unsupervised noise-reduction algorithms to automatically identify alternating character pairs.

Experimental Evaluation

The models were evaluated using the Malawi SMS Corpus, a real-world dataset of approximately 600 messages from a rural medical clinic, collected in partnership with FrontlineSMS:Medic. Messages were categorized into nine labels spanning administrative, request-based, and illness-specific (e.g., HIV, TB) classes. For cross-linguistic comparison, all messages were also translated into English.

Key Results

The integration of subword modeling led to dramatic performance increases for Chichewa classification. The final system achieved an average F-score gain of 0.206 over the word-sequence baseline.

Table 1: Performance comparison across medical labels for Chichewa and English.

As shown in Table 1, the subword-aware system more than doubled classification accuracy for six out of nine medical labels. Notably, while the English baseline was initially 25.7% more accurate than the Chichewa baseline, the final subword-aware system reduced this disparity to just 9.2%.

Figure 1: Learning curves showing F-score improvement as a function of training data percentage.

The learning curves in Figure 1 illustrate that the "Final" systems consistently outperform their respective baselines across varying amounts of training data. The Chichewa Final system's trajectory shows a clear trend toward matching the performance of the English baseline, highlighting the effectiveness of subword features in overcoming data sparsity.

Discussion and Conclusion

A key insight from this research is the "English bias" prevalent in NLP. Subword modeling provided massive boosts for Chichewa but only marginal gains (~5%) for English, explaining why such techniques are often overlooked in English-centric research. The study also found that "over-segmenting" in unsupervised models is beneficial, as downstream classifiers can learn to filter out irrelevant fragments.

While limited by a small dataset size, this work proves that for the "next billion" users of digital services, NLP must move beyond word-level representations to accommodate the linguistic diversity and orthographic noise inherent in global communication. Future work will explore more sophisticated character n-gram models and latent topical features to further enhance classification robustness.

Summary updated

Abstract

For millions of people in less resourced regions of the world, text messages (SMS) provide the only regular contact with their doctor. Classifying messages by medical labels supports rapid responses to emergencies, the early identification of epidemics and everyday administration, but challenges include text-brevity, rich morphology, phonological variation, and limited training data. We present a novel system that addresses these, working with a clinic in rural Malawi and texts in the Chichewa language. We show that modeling morphological and phonological variation leads to a substantial average gain of F=0.206 and an error reduction of up to 63.8% for specific labels, relative to a baseline system optimized over word-sequences. By comparison, there is no significant gain when applying the same system to the English translations of the same texts/labels, emphasizing the need for subword modeling in many languages. Language independent morphological models perform as accurately as language specific models, indicating a broad deployment potential.
BibTeX
@inproceedings{munro2010subword,
  author = {Robert Munro and Christopher D. Manning},
  title = {{Subword Variation in Text Message Classification}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2010},
}
Figure