Leveraging Social Annotation for Topic Language Model Adaptation

Youzheng Wu, Kazuhiko Abe, Paul R. Dixon, Chiori Hori, Hideki KashiokaInterspeech 2012 · OpenReview

Introduction

Language Models (LMs) are critical components of Automatic Speech Recognition (ASR) systems, responsible for predicting word sequences based on linguistic context. However, a significant challenge remains: general-purpose LMs often lack the domain-specific nuances required for specialized topics. For instance, the word "lead" carries distinct meanings in chemical engineering versus management.

To address this, researchers employ Topic Adaptation, tuning models to specific subjects. Traditional approaches include supervised adaptation, which is labor-intensive and expensive, or unsupervised adaptation (e.g., Latent Dirichlet Allocation), which often produces noisy, non-interpretable clusters. This paper explores a novel middle ground: leveraging "social annotations"—the human-curated category labels found on platforms like Yahoo! Answers—to build more effective and interpretable topic-specific LMs.

Methodology

The proposed framework utilizes the "folksonomy" or collective intelligence of web users to categorize training data. Instead of relying on algorithmic clustering, the researchers tap into existing hierarchical structures created by millions of users.

System architecture for topic-based language model adaptation using social annotations.

The process involves four primary stages:

  1. Classifier Training: A multi-class Support Vector Machine (SVM) is trained on the Yahoo! Answers Webscope Dataset, which contains a rich hierarchy of human-labeled topics.
  2. Topic Segmentation: This SVM classifier is used to partition a massive, unlabeled general-domain corpus into 24 distinct topic "buckets" (e.g., "Business & Finance," "Science & Mathematics").
  3. Model Construction: Individual n-gram language models are built for each of these 24 categories.
  4. Runtime Adaptation: During the transcription of new audio (such as a TED talk), the system identifies the most relevant topics and dynamically blends the generic LM with the corresponding topic-specific models.

Experimental Setup

The researchers evaluated their approach using the following datasets:

  • Social Data: Yahoo! Answers Webscope Dataset (6,000 pairs per category).
  • Evaluation Data: IWSLT-2011 TED ASR dataset.
  • LM Training Data: A combination of in-domain TED transcriptions (124K sentences) and a general-domain corpus of news and web text (115M sentences).
  • Validation Corpora: HUB4 and English Gigaword Fourth Edition.

Results and Discussion

The social annotation approach demonstrated significant improvements over both baseline and unsupervised methods. On the IWSLT-2011 TED test set, the proposed method achieved a Word Error Rate (WER) of 25.2%, an absolute improvement of 0.5% over the generic model and the standard LDA-based unsupervised approach (both at 25.7%).

Absolute Word Error Rate (WER) improvement for specific TED talks using the proposed social annotation method.

Beyond WER, the method significantly reduced "perplexity"—a measure of how well a model predicts a sample. The researchers observed a 14.8% relative improvement in perplexity on the Gigaword corpus. Furthermore, the use of social labels provides high interpretability; developers can clearly see which topics (e.g., "Arts & Humanities") are driving model performance, unlike the opaque clusters generated by unsupervised algorithms.

Conclusion

By leveraging the collective intelligence of web users, this research demonstrates that social annotations can serve as a powerful, cost-effective resource for language model adaptation. While the current implementation uses 24 top-level categories, future work could explore the finer granularity of the Yahoo! Answers hierarchy (over 1,200 categories) to provide even more precise adaptation. Additionally, improving the underlying SVM classifier—which currently operates at 65% accuracy—presents a clear path for further performance gains in speech recognition accuracy.

Summary updated

BibTeX
@inproceedings{wu2012leveraging,
  author = {Youzheng Wu and Kazuhiko Abe and Paul R. Dixon and Chiori Hori and Hideki Kashioka},
  title = {{Leveraging Social Annotation for Topic Language Model Adaptation}},
  booktitle = {Interspeech},
  year = {2012},
  url = {https://openreview.net/forum?id=7JEisncsZH_},
}
Figure