TL;DR
This paper analyzes a relation extraction system's performance on social media documents, finding a significant drop in accuracy compared to professionally edited text. The authors attribute this to the informal language and writing styles prevalent in social media, suggesting a need for more robust features tailored to this domain.
What Problem Does This Paper Address?
Understanding relationships between entities in text (e.g., "engine" is a "part of" "car") is crucial for many NLP tasks like sentiment analysis and question answering. While relation extraction systems have shown promise on structured text like news articles, their performance on the more informal and diverse landscape of social media remains a challenge. This paper investigates why a standard relation extraction system, effective on traditional news sources, falters when applied to social media data, specifically examining the impact of different writing styles.
Methods
The authors adapt a relation extraction system based on prior work by Zhou et al. (2005), which uses lexical features. Their system takes pairs of mentions within a sentence and classifies their relationship using a Support Vector Machine (SVM) implemented with libLinear. The key innovation here isn't a new algorithm, but rather the application and analysis of an existing system on a new, challenging domain. They use a proprietary tokenizer and the Stanford Parser for feature generation. Minor modifications to the original feature set include a different heuristic for selecting head words and replacing specific entity types with "Unknown" when they might indicate the relation itself.
Datasets
- ACE-2004 Corpus: Used as a baseline to validate the system's performance on more traditional text. It consists of broadcast news transcripts and newswire articles.
- J.D. Power and Associates (JDPA) Sentiment Corpus: The primary dataset for this study. It contains automotive and camera-related documents. The authors focus on the automotive subset (457 documents). This corpus is further broken down into:
- JDPA Power Steering blog: Professionally written by JDPA analysts.
- Blogspot: A general blogging platform.
- LiveJournal: A social media site with informal user-generated content.
Key Insights and Ideas
- Social Media is Different: The core finding is that the informal, diverse writing styles of social media (like blogs and LiveJournal) significantly degrade relation extraction performance compared to professionally edited content.
- Feature Mismatch: The features developed for news articles, which assume more formal grammar and vocabulary, are less effective on social media.
- Informal Language Hinders: The heavy use of pronouns, less descriptive mention phrases, and sentence-level variations in social media make it harder for the system to identify entity relationships.
- Training Data Skew: The JDPA-authored documents, while more formal, contain a higher density of relations and mention pairs within sentences. This can skew the training data, potentially biasing the classifier towards features specific to this more formal subset.
- Extra-Sentential Relations: A significant portion of relations in the JDPA corpus occur between sentences, which the current system, focused on intra-sentential relations, completely misses.
Key Results
- The system achieved an F-Measure of 50.6 on the ACE-2004 corpus, which is considered a reasonable baseline.
- On the JDPA Corpus overall, the F-Measure dropped to 40.5.
- Performance varied dramatically by source:
- JDPA authored documents: F-Measure of 49.9.
- Blogspot documents: F-Measure of 38.3.
- LiveJournal documents: F-Measure of 24.6.
- This represents a substantial 25-point F-Measure difference between the most formal (JDPA) and most informal (LiveJournal) sources.
- Approximately 29% of entity relations in the JDPA Corpus are extra-sentential and thus not detected by the system.
Limitations and Future Work
- Informal Language Features: The current features are not well-suited for the informal language, pronoun usage, and less descriptive mention phrases found in social media.
- Extra-Sentential Relations: The system's focus on intra-sentential relations leads to a significant loss of information.
- Tokenization/Parsing: The paper acknowledges that poor tokenization or parsing of social media text could also be a contributing factor to performance issues, separate from the classification itself.
Future work should explore:
- Developing new features that better capture the nuances of social media language.
- Extending relation extraction to handle relationships that span across sentences.
- Investigating the impact of improved tokenization and parsing specifically for social media data.
Summary updated