Classifying Message Board Posts with an Extracted Lexicon of Patient Attributes

Ruihong Huang, Ellen RiloffEMNLP 2013 · OpenReview

Abstract

In online medical communities, distinguishing between specific case descriptions and general inquiries is essential for effective information routing. This study presents a method for classifying veterinary message board posts by automatically harvesting a specialized lexicon of patient attributes (e.g., age, breed, gender) from unlabeled data. By integrating these attributes as features, we significantly improve classification performance, particularly for the minority "general question" category, achieving an overall accuracy of 96%.

Introduction

Online forums like the Veterinary Information Network (VIN) host a vast array of content, primarily divided into two categories: patient-specific case studies (e.g., "My 3-year-old Golden Retriever has a cough") and general medical inquiries (e.g., "What are the side effects of Clavamox?"). Accurately routing these posts is critical for platform organization. However, this task is complicated by significant vocabulary overlap—both post types use similar medical terminology—and a severe class imbalance, where approximately 90% of posts are case-specific.

The core insight of this research is that the presence or absence of a patient, rather than the medical topic itself, is the primary differentiator. Therefore, reliable detection of patient attributes is key to identifying case-specific content.

Methodology

The researchers developed a three-phase pipeline to transition from limited labeled data to a high-performance classifier.

Overview of the three-phase pipeline for lexicon induction and post classification.
  1. Patient Introductory (PI) Sentence Detection: Recognizing that patient details typically appear early in a post, an SVM was trained to identify these "introductory" sentences.
  2. Attribute Tagging & Lexicon Induction: A Conditional Random Field (CRF) tagger was trained on a small manually labeled dataset to recognize seven attribute types: Species/Breed, Name, Age, Gender, Weight, Possessor, and Disease/Symptom. To expand coverage, this tagger processed 10,000 unannotated posts. Frequently tagged words were compiled into a Veterinary Patient Attribute (VPA) Lexicon.
  3. Final Classification: A final SVM classifier utilized standard bag-of-words features alongside new features derived from the VPA Lexicon, such as the presence of known breed names or specific age-breed pairings.

The Role of Domain-Specific Jargon

A significant challenge in veterinary NLP is the prevalence of informal jargon and shorthand. Standard tools often fail to recognize terms like "m/n" (male neutered), "fs" (female spayed), or age descriptions like "1999 model." The automated lexicon induction process was vital for capturing this idiosyncratic vocabulary. Furthermore, the study found that a static lexicon harvested from a large unannotated corpus provided much broader coverage than the raw CRF tagger alone, particularly for rare breeds and varied shorthand.

Experimental Results

The inclusion of VPA Lexicon features led to substantial performance gains across all metrics, as shown in the table below.

Comparison of classification performance using unigram features versus patient attribute features.

The macro-averaged F-score improved from 81% (using binary unigrams) to 88% when combining unigrams with VPA Lexicon features. Most notably, the recall for the minority "general posts" class jumped from 54% to 79%. By becoming highly proficient at identifying patient descriptions, the system effectively identified general posts through the absence of these attributes. The final system achieved an overall accuracy of 96%.

Conclusion

This research demonstrates that harvesting specialized lexicons from unlabeled data can overcome challenges posed by class imbalance and domain-specific jargon in text classification. While the "Patient Introductory" sentence assumption is specific to medical contexts, the underlying strategy of lexicon induction could be generalized to other fields, such as legal or technical forums, where specific entity types define the nature of the discourse. Future work may explore more sophisticated filtering techniques to further refine the harvested lexicons.

Summary updated

BibTeX
@inproceedings{huang2013classifying,
  author = {Ruihong Huang and Ellen Riloff},
  title = {{Classifying Message Board Posts with an Extracted Lexicon of Patient Attributes}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2013},
  url = {https://openreview.net/forum?id=2WZzsUFt8w},
}
Figure