TL;DR
This paper investigates how well a state-of-the-art parser, trained on formal text, performs on informal, user-generated content from online discussion forums. They find a significant drop in performance and identify common linguistic phenomena in these forums that challenge parsers, proposing simple transformations to improve accuracy.
What Problem Does This Paper Address?
The rise of user-generated content (UGC) like blogs, wikis, and discussion forums presents a challenge for Natural Language Processing (NLP) tools. These tools are typically trained on carefully edited, formal text (like the Wall Street Journal corpus). Applying them directly to the messy, unedited language of UGC often leads to a performance degradation. This paper specifically addresses the gap in understanding how well a sophisticated phrase structure parser performs on this type of informal text and identifies the specific linguistic features that cause parsing errors.
Methods
The authors use the Berkeley parser, a latent variable PCFG (Probabilistic Context-Free Grammar) parser, as their primary tool. They evaluate its performance under three conditions:
- Automatic Tokenization: The parser handles word segmentation itself.
- Gold Tokenization: Pre-segmented words are provided, but spelling errors remain.
- Corrected Spelling: Both tokenization and spelling are corrected.
To evaluate the parser's output, they employ Sparseval, a metric designed for evaluating spoken language parsing, which is more robust to mismatches between parser output and gold-standard trees than standard metrics like evalb. They also perform a qualitative analysis of parser errors to identify problematic linguistic phenomena. Based on this analysis, they experiment with two types of transformations:
- Input Sentence Transformations: Modifying the raw text before parsing (e.g., converting "cos" to "because").
- Treebank Transformations: Modifying the parser's training data to better reflect the characteristics of the target domain.
Datasets
- Training Data: The standard Wall Street Journal (WSJ) corpus is used to train the Berkeley parser.
- Evaluation Data:
- Development Set: 42 posts (185 sentences) from a BBC Sport 606 Football discussion forum thread about a refereeing decision.
- Test Set: 40 posts (170 sentences) from a BBC Sport 606 Football discussion forum thread about a player's behavior.
- Gold Standard: Hand-created phrase structure trees for the sentences in both the development and test sets. Tokenization and spelling correction were also done manually for these sets.
Key Insights and Ideas
- Informal Text is Hard: Even a powerful parser trained on formal data struggles significantly with user-generated content. The performance drop is substantial.
- Spelling Errors are Less of a Problem Than Expected: Surprisingly, the parser's mechanism for handling unknown words (using suffix information) makes it relatively robust to many content word spelling errors. Function word errors, however, can be more problematic.
- Tokenization Matters: When the parser has to perform its own tokenization on informal text, performance suffers. This is partly due to the under-use of apostrophes, leading to words like "didnt" being misclassified.
- Specific Phenomena Cause Trouble: The paper identifies several key linguistic features in discussion forums that lead to parser errors, including:
- Omission of Conjunctions: Sentences are often joined by commas or simply juxtaposed without explicit conjunctions.
- Unusual Coordination: Coordination of constituents that are not grammatically parallel.
- Idioms and Fixed Expressions: Phrases whose meaning isn't compositional.
- Acronyms and Abbreviations: Like "lmao" or "cos."
- Capitalization Issues: Excessive use of CAPS LOCK.
- Simple Transformations Help: Basic input sentence and treebank transformations can significantly improve parsing performance on this type of data.
Key Results
- The baseline Berkeley parser (SM5 grammar) achieved a Parseval f-score of 77.56% on the development set.
- Applying simple input sentence and treebank transformations increased the f-score to 80.27% on the development set.
- The performance drop from WSJ to the football forum data was more pronounced than previously reported for other parsers on similar tasks.
- Transformations targeting lexical rules (e.g., adverb endings, proper noun capitalization) and input sentence normalization (e.g., "cos" to "because") were most effective.
Limitations and Future Work
- Small Data Size: The authors acknowledge that the development set is small, making it difficult to draw definitive conclusions about complex phenomena like coordination and subject ellipsis.
- Domain Specificity: The findings are specific to the BBC Sport forum data. Further research is needed to see if these results generalize to other types of UGC.
- Future Work:
- More thorough exploration of treebank transformations, potentially using data from other sources like the Switchboard corpus.
- Investigating more sophisticated methods for handling the identified problematic phenomena.
- Exploring domain adaptation techniques beyond simple transformations, such as self-training.
Summary updated