From News to Comment: Resources and Benchmarks for Parsing the Language of Web 2.0

Jennifer Foster, Özlem Çetinoglu, Joachim Wagner, Joseph Le Roux, Joakim Nivre, Deirdre Hogan, Josef van GenabithIJCNLP 2011 · OpenReview

Abstract

Jennifer Foster, Özlem Çetinoğlu, Joachim Wagner, Joseph Le Roux, Joakim Nivre, Deirdre Hogan, Josef van Genabith. Proceedings of 5th International Joint Conference on Natural Language Processing.

2011.

Introduction

Most state-of-the-art natural language parsers are trained on the Penn Treebank, which consists of highly grammatical, edited news text from the Wall Street Journal (WSJ) in the late 1980s. When these parsers encounter "Web 2.0" language—characterized by slang, hashtags, URLs, non-standard capitalization, and "agrammatical" fragments—their performance plummets. This paper introduces a new gold-standard dataset of 1,000 hand-corrected parse trees for social media text (tweets and forum posts) to benchmark how well "news-trained" parsers handle the chaos of Web 2.0 and investigates unsupervised domain adaptation techniques to bridge the performance gap.

The Web 2.0 Treebank

To measure the performance gap, the researchers developed the Web 2.0 Treebank, consisting of 1,000 hand-corrected trees split between Twitter (519 sentences) and BBC Football forums (481 sentences). This resource allows for a rigorous evaluation of parsers against the unique challenges of microblogs and discussion forums, which often act as a "middle ground" between formal news and the extreme informality of tweets.

Baseline Performance and the "Twitter Gap"

The researchers evaluated four popular statistical parsers: Berkeley (PCFG-LA), Brown (reranking), Malt (transition-based), and MST (graph-based). The results revealed a stark "baseline drop." While these parsers achieve approximately 90% F-score on standard news (WSJ), performance falls to roughly 79% on forum posts and a dismal ~71% on tweets.

Baseline performance of four parsers across WSJ, Football, and Twitter domains.

Twitter is identified as the "final boss" of parsing due to the extreme density of unknown tokens like URLs, usernames, and slang. A significant bottleneck is Part-of-Speech (POS) tagging; mislabeling a slang word like "grrr" can cause the entire tree structure to collapse. Providing "gold" POS tags to the Berkeley parser improved its Twitter F-score by over 5 points, highlighting the sensitivity of structural analysis to lexical noise.

Domain Adaptation: Self-Training and Uptraining

To recover performance without requiring massive new labeled datasets, the authors experimented with three semi-supervised retraining strategies using large unlabeled corpora (TwitterTrain and FootballTrain):

  • Self-Training: Using a high-quality reranking parser (Brown) to label millions of unlabeled social media sentences, then adding those "silver" trees back into the training set.
  • Uptraining: Using a more accurate but slower phrase-structure parser to generate training data for a faster, less accurate dependency parser (Malt).
  • PCFG-LA Self-Training: An iterative approach for the Berkeley-style parser using latent variable modeling to prevent overfitting to its own mistakes.
Evalb f-score on Twitter development data as the size of the unlabeled Twitter training set increases.

Key Results and Insights

Retraining yielded meaningful gains, though the difficulty of the domain dictated the level of success. For the Malt parser, retraining on forum data led to a massive 8.8% absolute increase in Labelled Attachment Score (LAS). For the Brown parser, self-training on forum data improved the F-score by 2.4%.

Improvements on Twitter were more modest (around 1.2% to 1.7%), as shown in the learning curves where performance gains often plateau despite increasing the amount of unlabeled data. This suggests that while self-training helps, microblogging remains a significantly tougher challenge than discussion forums due to its inherent linguistic volatility.

Conclusion

The study demonstrates that while standard parsers struggle with Web 2.0 text, unsupervised domain adaptation can meaningfully recover performance. Future work may focus on text normalization (e.g., "u" → "you") and ensemble methods that combine the strengths of different parser architectures to better handle the noise of social media.

Summary updated

BibTeX
@inproceedings{foster2011from,
  author = {Jennifer Foster and Özlem Çetinoglu and Joachim Wagner and Joseph Le Roux and Joakim Nivre and Deirdre Hogan and Josef van Genabith},
  title = {{From News to Comment: Resources and Benchmarks for Parsing the Language of Web 2.0}},
  booktitle = {IJCNLP},
  year = {2011},
  url = {https://openreview.net/forum?id=klX__PvrzRp},
}
Figure