The French Social Media Bank: a Treebank of Noisy User Generated Content

Djamé Seddah, Benoît Sagot, Marie Candito, Virginie Mouilleron

TL;DR

This paper introduces the French Social Media Bank, the first treebank specifically designed for noisy, user-generated French text from social media. It addresses the significant performance drop of NLP tools on such data and provides a benchmark for future research in this challenging area.

What Problem Does This Paper Address?

Modern NLP models, particularly statistical parsers, achieve impressive results on well-edited texts like news articles. However, their performance plummets when applied to "noisy" user-generated content (UGC) from social media platforms. This UGC deviates significantly from standard linguistic, spelling, and typographic conventions, making it difficult for existing tools to process. This gap hinders valuable applications like data mining and sentiment analysis on the vast amount of text generated online. The paper aims to bridge this gap by creating a dedicated resource and evaluating current parsing capabilities on this challenging data.

Methods

The authors developed a novel annotation scheme and methodology to create the French Social Media Bank. Key aspects include:

  • Noisiness Measurement: An ad-hoc metric based on character trigram distributions was developed to quantify the divergence of UGC from standard French text. This allowed them to categorize subcorpora by their level of "noisiness."
  • Annotation Scheme: The scheme is largely based on the existing French Treebank (FTB) guidelines but extends them to handle UGC-specific phenomena. This includes specific guidelines for contractions, typographic diaeresis, and meta-language elements like hashtags and mentions.
  • Pre-annotation Strategies: Two strategies were employed based on the noisiness score:
  • For less noisy corpora, an extended version of standard FTB tools was used.
  • For highly noisy corpora, a specialized pre-annotation process was developed. This involved using regular expressions to identify entities (URLs, hashtags, etc.), a custom tokenizer, and a set of rewriting rules to "correct" tokens before applying a POS tagger trained on FTB data.
  • Constituency and Functional Annotation: This was performed using a state-of-the-art statistical parser, with manual validation and correction by annotators.

Datasets

The primary dataset introduced is the French Social Media Bank, containing approximately 1,700 sentences from various Web 2.0 sources, including:

  • Doctissimo: A health forum.
  • JeuxVideos.com: A video game forum.
  • Twitter: Microblogging platform.
  • Facebook: Social networking platform.

The paper also references the French Treebank (FTB) as a reference corpus for measuring noisiness and as the basis for training data for their evaluation experiments.

Key Insights and Ideas

  • "Noisy" is a Spectrum: The paper demonstrates that UGC isn't uniformly noisy. They successfully quantify and categorize different levels of noise, which is crucial for developing targeted processing strategies.
  • Standard NLP Tools Struggle Dramatically: The results clearly show that even with adaptations, parsing performance on UGC is significantly lower than on edited texts, highlighting the fundamental challenges posed by non-standard language.
  • Morphologically Rich Languages Add Complexity: French, being a morphologically rich language, exacerbates the parsing difficulty when dealing with noisy UGC, as variations in spelling and morphology are more impactful.
  • Dedicated Resources are Essential: The creation of the French Social Media Bank underscores the need for specialized datasets to train and evaluate NLP models for real-world, informal language.

Key Results

  • POS Tagging: The normalization and correction wrapper significantly improved POS tagging accuracy, with results on standard Twitter subcorpora being comparable to previous work on English Twitter data. Accuracy scores ranged from the mid-50s to the high-80s depending on the subcorpus and the use of the correction wrapper.
  • Statistical Parsing: Baseline parsing results were considerably lower than on edited texts. F1 scores for labeled brackets ranged from a low of 39.11% for the noisiest subcorpora to around 71-72% for better-edited web parts. This is in stark contrast to the 86-89% typically achieved on the FTB.
  • Facebook vs. Twitter: The parser performed worse on Facebook data than on Twitter data, which the authors attribute to Twitter data being more homogeneous due to its microblogging nature, while Facebook posts are more distributed.

Limitations and Future Work

  • Limited Scale: The first release of the French Social Media Bank is relatively small (1,700 sentences).
  • Baseline Performance: The reported parsing results are baselines, indicating significant room for improvement.
  • Future Work: The authors intend to apply their normalization and correction module before parsing to provide the parser with corrected tokens that more closely resemble their training data. They also plan to explore self-training techniques and unsupervised word clustering on larger unlabeled corpora for further improvements. The ultimate goal is to develop robust NLP tools for processing user-generated content in French.

Summary updated

Abstract

In recent years, statistical parsers have reached high performance levels on well-edited texts. Domain adaptation techniques have improved parsing results on text genres differing from the journalistic data most parsers are trained on. However, such corpora usually comply with standard linguistic, spelling and typographic conventions. In the meantime, the emergence of Web 2.0 communication media has caused the apparition of new types of online textual data. Although valuable, e.g., in terms of data mining and sentiment analysis, such user-generated content rarely complies with standard conventions: they are noisy. This prevents most NLP tools, especially treebank based parsers, from performing well on such data. For this reason, we have developed the French Social Media Bank, the first user-generated content treebank for French, a morphologically rich language (MRL). The first release of this resource contains 1,700 sentences from various Web 2.0 sources, including data specifically chosen for their high noisiness. We describe here how we created this treebank and expose the methodology we used for fully annotating it. We also provide baseline POS tagging and statistical constituency parsing results, which are lower by far than usual results on edited texts. This highlights the high difficulty of automatically processing such noisy data in a MRL.
BibTeX
@inproceedings{seddah2012french,
  author = {Djamé Seddah and Benoît Sagot and Marie Candito and Virginie Mouilleron},
  title = {{The French Social Media Bank: a Treebank of Noisy User Generated Content}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2012},
  url = {https://openreview.net/forum?id=C2SPM2RV__n},
}
Figure