Language Modeling for Voice-Enabled Social TV Using Tweets

Junlan Feng, Bernard Renger

Abstract

This study explores the enhancement of Automatic Speech Recognition (ASR) for "Social TV" applications by leveraging Twitter data. By developing a specialized text-normalization pipeline to address the linguistic noise inherent in social media—such as hashtags, abbreviations, and typos—researchers achieved a 3.7% absolute improvement in word accuracy. The findings demonstrate that bridging the gap between "Twitter-speak" and spoken English is critical for effective voice-enabled social interfaces.

Introduction

"Social TV"—the practice of interacting with others on social media while watching television—has become a significant trend in media consumption. However, the interface for these interactions remains a bottleneck. Typing on remotes or small mobile screens while simultaneously watching a primary screen is cumbersome. Voice-enabled interfaces, powered by Automatic Speech Recognition (ASR), offer a more natural alternative, allowing users to speak their updates.

The Challenge of Twitter Data

ASR systems rely on Language Models (LMs) to predict word sequences. To be effective, these models require vast amounts of data that reflect the specific domain of discourse. Twitter provides a rich source of TV-related conversation, but it is notoriously "noisy." Tweets are characterized by hashtags, URLs, intentional misspellings (e.g., "yesssss"), and abbreviations (e.g., "2nite"). Training a model on raw Twitter data often leads to poor performance because spoken language typically follows more standard grammatical conventions.

Methodology: Bridging the Gap

The researchers developed a multi-stage pipeline to transform messy Twitter text into a format suitable for training high-performing ASR models.

Tweet Filtering

A machine learning classifier was employed to identify relevant TV-related tweets. This is a non-trivial task; for instance, a search for the show House might yield results about real estate. The classifier achieved 84.3% precision in identifying actual TV content.

Normalization Phases

The core of the approach involves two normalization steps:

  • Phase-I (Cleaning): This stage removes elements that are rarely spoken, such as usernames (@mentions), hashtags (#), and URLs.
  • Phase-II (Correction): This stage addresses linguistic noise. Intentional misspellings are reduced (e.g., "aaaawwww" to "aw"), and abbreviations are mapped to spoken equivalents (e.g., "lol" to "L O L"). A RandomForest classifier was used to correct unintentional typos by ranking candidates based on lexical and phonetic similarity.

Language Model Interpolation

The final Social TV language model was created by interpolating a general-purpose LM with the models trained on the cleaned Twitter data.

Experimental Setup

The system was evaluated using a baseline model trained on a mix of broadcast news, Q&A queries, spoken SMS, and web searches. The Twitter corpus consisted of 3 million TV-related tweets collected between September 2011 and March 2012. For testing, a custom mobile app was used to collect 483 spoken utterances from real users discussing 32 popular TV shows, which were then manually transcribed to provide a ground truth.

Key Results

The performance was measured using Word Accuracy (WACC). The results demonstrate that while basic cleaning (Phase-I) provides a significant boost, advanced correction (Phase-II) offers additional incremental gains.

Table 1: Word Accuracy (WACC) comparison across different language models.

The transition from the baseline model (60.1%) to the Phase-II normalized model (63.8%) represents a 3.7% absolute improvement. In the field of ASR, where gains are often hard-won, this is a substantial advancement.

Conclusion

The study highlights that the primary barrier to using social media data for ASR is the stylistic mismatch between typed and spoken language. By implementing a sophisticated normalization pipeline that accounts for phonetic similarity and removes non-spoken elements, researchers can effectively leverage "noisy" data sources like Twitter to enhance voice-enabled user experiences. Future work will explore show-specific models and more advanced typo classification features.

Summary updated

BibTeX
@inproceedings{feng2012language,
  author = {Junlan Feng and Bernard Renger},
  title = {{Language Modeling for Voice-Enabled Social TV Using Tweets}},
  booktitle = {Interspeech},
  year = {2012},
  doi = {10.21437/interspeech.2012-616},
}
Figure