TL;DR
This paper presents a surprisingly simple yet highly effective method for classifying text into fictional and non-fictional genres using just two linguistic features: the ratio of adverbs to adjectives and the ratio of adjectives to pronouns. These features, identified through a rigorous feature selection process, achieve classification accuracy on par with models using a much larger set of features, offering both practical applications and linguistic insights.
What Problem Does This Paper Address?
Distinguishing between fictional and non-fictional text is a fundamental task with applications ranging from content filtering and recommendation systems to understanding stylistic nuances. While previous approaches have relied on a broad spectrum of linguistic features (lexical, syntactic, character-level, and derived metrics), this paper questions whether a simpler set of features could achieve comparable or even superior performance. The authors aim to identify the most salient linguistic indicators that differentiate imaginative prose from matter-of-fact writing, moving beyond purely algorithmic performance to gain insights into the underlying linguistic properties of these genres.
Methods
The core of this research lies in a two-stage feature selection process using Recursive Feature Elimination with Cross-Validation (RFECV).
- Initial Feature Extraction: The researchers extracted a comprehensive set of nineteen linguistic features from the Brown Corpus, encompassing low-level (e.g., average sentence length, type-token ratio) and high-level (e.g., counts of different parts of speech) features.
- Feature Selection:
- Gold Standard Tags: RFECV was applied to features derived from gold-standard part-of-speech (POS) tags. This process identified six highly significant features.
- Automatic Tags: Crucially, the authors then applied RFECV to features derived from automatically tagged text (using the Stanford tagger). This step, which accounts for potential errors in automatic tagging, revealed that only two features – the adverb/adjective ratio and the adjective/pronoun ratio – were most significant.
- Classification: A logistic regression classifier was employed for the binary classification task (fiction vs. non-fiction). This choice is motivated by logistic regression's ability to provide not only classification but also insights into feature relevance and directionality.
Datasets
The study utilized two primary datasets:
- Brown Corpus: A foundational corpus of present-day American English, widely used in linguistic research. It was used for both feature selection and training/testing.
- British National Corpus (BNC) Baby: A smaller corpus of British English, used to test the generalizability of the model and its applicability to different English varieties.
The authors also created a subset of the Brown Corpus by excluding the "news" category to investigate the model's robustness to specific subgenres.
Key Insights and Ideas
- Simplicity Trumps Complexity: The most striking insight is that a model using just two carefully selected linguistic features (adverb/adjective ratio and adjective/pronoun ratio) performs as well as, and sometimes better than, models using a much larger set of nineteen features. This challenges the common assumption that more features always lead to better performance.
- Linguistic Significance of Ratios: The identified features are not arbitrary counts but ratios that capture specific stylistic tendencies. Fiction tends to use more adverbs relative to adjectives (perhaps for descriptive flair), while non-fiction uses more adjectives relative to pronouns (potentially for more direct description).
- Robustness Across Corpora: The model's strong performance on both American (Brown) and British (BNC Baby) English suggests that these identified features are robust and not heavily influenced by dialectal variations.
- Feature Selection Matters: The difference in outcomes between using gold-standard and automatically-tagged data highlights the importance of robust feature selection methods, especially when dealing with imperfect automated linguistic analysis. The RFECV process effectively filtered out noise introduced by automatic tagging errors.
Key Results
- High Accuracy with Two Features: The two-feature model (adverb/adjective ratio and adjective/pronoun ratio) achieved classification accuracies of 96.31% on the Brown Corpus and 97.13% on a combined Brown/BNC dataset, with a standard deviation of less than 1%.
- On Par with Exhaustive Sets: These accuracies are comparable to or exceed those achieved by models using all nineteen features or six selected features.
- Universal Applicability: When testing on the BNC Baby corpus after training on the Brown Corpus, the two-feature model achieved 100% classification accuracy, demonstrating its strong generalizability.
- Feature Importance: The logistic regression coefficients indicated that the adverb/adjective ratio had a positive association with fiction, while the adjective/pronoun ratio had a negative association (i.e., higher for non-fiction).
Limitations and Future Work
- Minimum Text Length: The paper acknowledges that the minimum text length required for accurate classification remains an open question.
- Larger Datasets: The authors suggest that future work should involve studying a much larger dataset to further validate the efficacy of their identified features.
- Generalizability: While the study shows promise across two English corpora, further investigation into the generalizability of these features to other languages or more diverse text types would be valuable.
- Qualitative Analysis: While the paper provides quantitative results, a deeper qualitative analysis of how these specific ratios manifest in actual text could offer richer linguistic insights.
Summary updated