TL;DR
This paper introduces a simple yet effective method for automatically scoring words and phrases based on their stylistic properties, specifically formality and complexity. By analyzing how often phrases appear in contrasting corpora, they can generate style scores that correlate well with human judgments and are useful for tasks like identifying stylistic differences in paraphrases and characterizing genre variations.
What Problem Does This Paper Address?
Understanding "how" something is said, beyond just "what" is said, is crucial for advanced language technologies. While lexical features are vital for many NLP tasks, research on stylistic variation at the lexical level (individual words and short phrases) has been relatively scarce compared to studies on larger text units. This paper aims to bridge this gap by developing a method to quantify stylistic properties of words and phrases, enabling finer-grained analysis of language. This is important for applications like tailored search, context-aware inference, and author belief analysis, where subtle stylistic cues can significantly impact meaning and perception.
Methods
The core of the method involves calculating a "style score" for phrases based on their occurrence rates in stylistically contrasting corpora. For formality, they compare the phrase's probability in a formal corpus (Europarl) versus a combined corpus (Europarl + Switchboard). For complexity, they compare a standard Wikipedia corpus against a simplified Wikipedia corpus. The score is a log-ratio: log(P(phrase | Reference Corpus) / P(phrase | All Corpora)). This approach is designed to be simple to implement and scalable. They then evaluate these scores intrinsically by comparing them to human judgments and extrinsically in applications like paraphrase differentiation and genre analysis.
Datasets
- Europarl corpus: Used as an example of formal text.
- Switchboard corpus: Used as an example of casual text.
- Wikipedia corpus: Used as an example of complex language.
- Simplified Wikipedia corpus: Used as an example of simple language.
- Paraphrase Database (PPDB): Used for identifying paraphrase pairs with stylistic variations.
- MASC corpus: Used for sentence-level human judgments of formality and complexity across 18 genres.
Key Insights and Ideas
- Lexical style is quantifiable: The paper demonstrates that stylistic properties like formality and complexity can be effectively captured by analyzing word/phrase frequencies across different corpora.
- Simplicity and scalability: The log-ratio method is intuitive and computationally efficient, making it suitable for large-scale applications.
- Formality and complexity are distinct but correlated: While the paper focuses on these two dimensions, they observe a significant correlation between them in human perception.
- Style is often independent of topic: The "PP only" method, which focuses on paraphrases, shows that stylistic choices can be identified even when the underlying topic is the same, highlighting the importance of lexical choice.
- Human agreement varies: While inter-annotator agreement is generally good, it's lower at the sentence level than at the phrase level, suggesting that context and topic can sometimes obscure lexical style for human annotators.
Key Results
- Intrinsic Evaluation: The automatically induced style scores show strong correlation with human judgments. For formality, the correlation matches inter-human levels (Spearman ρ ≈ 0.65). For complexity, the correlation is also significant (Spearman ρ ≈ 0.44).
- Paraphrase Differentiation: The method can accurately identify paraphrase pairs with stylistic differences. High-confidence predictions for complexity achieved 90% accuracy, and for formality, 88% accuracy.
- Genre Characterization: The "all words" method for genre characterization correlates well with human judgments (Spearman ρ ≈ 0.77 for formality, 0.80 for complexity). Even the "PP only" method, which controls for topic, maintains significant correlations (Spearman ρ ≈ 0.63 for formality, 0.45 for complexity).
Limitations and Future Work
- Topic vs. Style: The paper acknowledges that the "all words" method can conflate topic and style, as certain topics naturally lend themselves to more formal or complex language. Future work aims to better disentangle these influences.
- Automating Paraphrase Equivalence: The current evaluation of paraphrase pairs relies on human judgment to determine if they are stylistically equivalent. Automating this identification is a future research direction.
- Sentence-level Agreement: The lower inter-annotator agreement at the sentence level suggests a need for further exploration into how humans separate style from topic and how lexical choices influence this distinction.
- Web-scale Style Annotation: The authors see their method as a promising approach for incorporating nuanced linguistic information into language applications, suggesting future work in building more comprehensive and nuanced stylistic lexicons.
Summary updated