TL;DR
This paper introduces the French Social Media Bank, the first treebank specifically designed for noisy, user-generated French text from social media. It addresses the significant performance drop of NLP tools on such data and provides a benchmark for future research in this challenging area.
What Problem Does This Paper Address?
Modern NLP models, particularly statistical parsers, achieve impressive results on well-edited texts like news articles. However, their performance plummets when applied to "noisy" user-generated content (UGC) from social media platforms. This UGC deviates significantly from standard linguistic, spelling, and typographic conventions, making it difficult for existing tools to process. This gap hinders valuable applications like data mining and sentiment analysis on the vast amount of text generated online. The paper aims to bridge this gap by creating a dedicated resource and evaluating current parsing capabilities on this challenging data.
Methods
The authors developed a novel annotation scheme and methodology to create the French Social Media Bank. Key aspects include:
- Noisiness Measurement: An ad-hoc metric based on character trigram distributions was developed to quantify the divergence of UGC from standard French text. This allowed them to categorize subcorpora by their level of "noisiness."
- Annotation Scheme: The scheme is largely based on the existing French Treebank (FTB) guidelines but extends them to handle UGC-specific phenomena. This includes specific guidelines for contractions, typographic diaeresis, and meta-language elements like hashtags and mentions.
- Pre-annotation Strategies: Two strategies were employed based on the noisiness score:
- For less noisy corpora, an extended version of standard FTB tools was used.
- For highly noisy corpora, a specialized pre-annotation process was developed. This involved using regular expressions to identify entities (URLs, hashtags, etc.), a custom tokenizer, and a set of rewriting rules to "correct" tokens before applying a POS tagger trained on FTB data.
- Constituency and Functional Annotation: This was performed using a state-of-the-art statistical parser, with manual validation and correction by annotators.
Datasets
The primary dataset introduced is the French Social Media Bank, containing approximately 1,700 sentences from various Web 2.0 sources, including:
- Doctissimo: A health forum.
- JeuxVideos.com: A video game forum.
- Twitter: Microblogging platform.
- Facebook: Social networking platform.
The paper also references the French Treebank (FTB) as a reference corpus for measuring noisiness and as the basis for training data for their evaluation experiments.
Key Insights and Ideas
- "Noisy" is a Spectrum: The paper demonstrates that UGC isn't uniformly noisy. They successfully quantify and categorize different levels of noise, which is crucial for developing targeted processing strategies.
- Standard NLP Tools Struggle Dramatically: The results clearly show that even with adaptations, parsing performance on UGC is significantly lower than on edited texts, highlighting the fundamental challenges posed by non-standard language.
- Morphologically Rich Languages Add Complexity: French, being a morphologically rich language, exacerbates the parsing difficulty when dealing with noisy UGC, as variations in spelling and morphology are more impactful.
- Dedicated Resources are Essential: The creation of the French Social Media Bank underscores the need for specialized datasets to train and evaluate NLP models for real-world, informal language.
Key Results
- POS Tagging: The normalization and correction wrapper significantly improved POS tagging accuracy, with results on standard Twitter subcorpora being comparable to previous work on English Twitter data. Accuracy scores ranged from the mid-50s to the high-80s depending on the subcorpus and the use of the correction wrapper.
- Statistical Parsing: Baseline parsing results were considerably lower than on edited texts. F1 scores for labeled brackets ranged from a low of 39.11% for the noisiest subcorpora to around 71-72% for better-edited web parts. This is in stark contrast to the 86-89% typically achieved on the FTB.
- Facebook vs. Twitter: The parser performed worse on Facebook data than on Twitter data, which the authors attribute to Twitter data being more homogeneous due to its microblogging nature, while Facebook posts are more distributed.
Limitations and Future Work
- Limited Scale: The first release of the French Social Media Bank is relatively small (1,700 sentences).
- Baseline Performance: The reported parsing results are baselines, indicating significant room for improvement.
- Future Work: The authors intend to apply their normalization and correction module before parsing to provide the parser with corrected tokens that more closely resemble their training data. They also plan to explore self-training techniques and unsupervised word clustering on larger unlabeled corpora for further improvements. The ultimate goal is to develop robust NLP tools for processing user-generated content in French.
Summary updated