The French Social Media Bank: a Treebank of Noisy User Generated Content

Djamé Seddah, Benoît Sagot, Marie Candito, Virginie Mouilleron, Vanessa Combet

Introduction

Natural Language Processing (NLP) tools, such as part-of-speech (POS) taggers and syntactic parsers, are typically trained on high-quality, edited text. In the French context, the standard training set is the French Treebank (FTB), derived from the journalistic prose of Le Monde. However, these tools often fail when applied to the "wild" language of social media—User Generated Content (UGC). This paper introduces the French Social Media Bank (FSMB), the first gold-standard treebank designed to address the challenges of noisy French UGC.

The Challenge of Noisy UGC

Social media text is characterized by "noise": phonetic misspellings, slang, missing punctuation, and non-standard grammar. For Morphologically Rich Languages (MRLs) like French, this noise is particularly disruptive. A single typo can alter the perceived gender, number, or tense of a word, causing a cascade of errors in syntactic parsing. The authors quantify this gap and propose a resource to bridge it.

Quantifying Noise

The researchers sampled approximately 1,700 sentences from four sources: Facebook, Twitter, Doctissimo (a health forum), and JeuxVideos.com. To measure the "noise" level, they used an ad-hoc metric based on character trigrams, comparing the social media subcorpora to the standard FTB.

Noisiness scores across different subcorpora compared to the standard French Treebank (FTB).

As shown in the data, noise is not uniform across platforms. Facebook comments were found to be significantly noisier than Twitter posts, likely due to Twitter's character limits and its role as a broadcast medium, which encourages more standardized writing.

Handling Noisy Tokens

The FSMB annotation scheme (FTB-UC) adapts standard guidelines to handle UGC-specific phenomena. The researchers categorized non-standard tokens into three main types: ergographic (misspellings), transverse (contractions or splits), and marks of expressiveness (emoticons and stretching).

Examples of linguistic phenomena found in the French Social Media Bank.

To improve baseline performance, the authors developed a normalization wrapper—a pre-processing pipeline using 327 rewriting rules to "translate" noisy tokens into standard French before tagging, then mapping those tags back to the original text.

Key Results

The study reveals a dramatic "performance cliff." While parsers achieve 84-86% F-scores on standard FTB text, performance drops to as low as 39.11% on the noisiest UGC sub-corpora. However, the normalization wrapper proved essential: POS tagging accuracy on high-noise Doctissimo data jumped from 56.41% to 80.78%. Across the entire FSMB, the average parsing F-score was 67.58%, confirming that UGC is significantly harder to process than even complex domain-specific text like biomedical literature.

Conclusion

The French Social Media Bank provides a critical evaluation set for French NLP. While its size (1,700 sentences) is modest, it serves as a high-quality "seed" for future semi-supervised learning approaches. By guiding models with this gold-standard data, researchers can move toward building parsers that are robust enough for the diverse linguistic landscape of the modern web.

Summary updated

BibTeX
@inproceedings{seddah2012french,
  author = {Djamé Seddah and Benoît Sagot and Marie Candito and Virginie Mouilleron and Vanessa Combet},
  title = {{The French Social Media Bank: a Treebank of Noisy User Generated Content}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2012},
  url = {https://openreview.net/forum?id=C2SPM2RV__n},
}
Figure