TL;DR
This paper investigates which types of text features (style, content, or hybrid) are most effective for authorship attribution across different datasets. They found that the optimal feature choice is highly dependent on the dataset's characteristics, particularly the topical similarity between authors, and developed a method to predict this. Their findings led to a new state-of-the-art performance on two benchmark datasets.
What Problem Does This Paper Address?
Authorship attribution, the task of identifying an author from their writing, has traditionally relied on analyzing either an author's unique writing style (stylometry) or the topics they tend to write about. While many features have been proposed, there's been a lack of systematic analysis on how dataset characteristics influence the effectiveness of these features. This makes it difficult for researchers to choose the best features for a given task, potentially leading to suboptimal performance. The authors aim to bridge this gap by exploring the relationship between dataset properties and feature effectiveness, providing a more principled approach to feature selection for authorship attribution.
Methods
The core of this research involves a two-pronged approach:
- Dataset Analysis: The authors use Latent Dirichlet Allocation (LDA) to model the topic distributions within each dataset. By calculating the Jensen-Shannon Divergence (JSD) between author topic distributions, they quantify the topical similarity (or dissimilarity) among authors in each dataset. This provides a crucial understanding of the dataset's inherent topical structure.
- Feature Ablation Study: They then conduct experiments using two classifiers (Feed-forward Neural Network and Logistic Regression) on four established authorship attribution datasets. They systematically remove different feature types (style, content, and hybrid) to measure their individual impact on attribution accuracy. This "ablation" process reveals which features are most critical for each dataset.
The paper also proposes an extension to a previously developed neural network model (Sari et al., 2017) by incorporating the identified optimal feature types as auxiliary inputs.
Datasets
The study utilizes four widely used authorship attribution datasets:
- Judgment: Legal judgments from three Australian High Court judges.
- CCAT10: A subset of Reuters Corpus Volume 1 (RCV1).
- CCAT50: A larger subset of Reuters Corpus Volume 1 (RCV1).
- IMDb62: Movie reviews and message board posts from the Internet Movie Database.
These datasets were chosen for their common use in prior literature and their diverse characteristics in terms of author count, topic/genre, and document length.
Key Insights and Ideas
- Dataset Characteristics Dictate Feature Effectiveness: The most significant finding is that there's no one-size-fits-all feature set for authorship attribution. The optimal choice is strongly tied to the dataset's topical diversity.
- Content Features Shine in Diverse Topics: For datasets where authors discuss a wide range of topics (like news articles in CCAT10/CCAT50), content-based features (like word n-grams) are generally more effective. This is because topical differences are a strong signal for author identification.
- Style Features Excel in Homogeneous Topics: Conversely, for datasets where authors tend to discuss similar topics (like legal judgments in Judgment or movie reviews in IMDb62), style-based features (like function word usage, punctuation) become more crucial.
- Character N-grams Aren't Always King: Contrary to common practice, character n-grams (often considered a go-to feature for their ability to capture both style and topic) do not perform equally well across all datasets. Their effectiveness is context-dependent.
- Predicting Optimal Features: The study demonstrates that by analyzing dataset topical similarity (using LDA), one can predict which feature type will yield the best results for that specific dataset.
Key Results
- The authors' analysis successfully predicted the most effective features for three out of the four datasets.
- In feature ablation experiments, content-based features were found to be most effective for the CCAT datasets, while style-based features were more beneficial for the Judgment and IMDb62 datasets.
- Their extended neural model, incorporating the identified optimal feature types, achieved new state-of-the-art performance on the CCAT50 and IMDb62 datasets. Specifically, the improvement on IMDb62 was statistically significant (p < 0.05).
Limitations and Future Work
- Judgment Dataset Anomaly: The study noted that style features did not improve results on the Judgment dataset as expected. They hypothesize this might be because the baseline continuous character n-gram model already effectively captured the authors' style, leaving little room for improvement from auxiliary style features.
- Future Work: The paper suggests that this principled approach to feature selection can be further validated and applied to other domains and tasks beyond authorship attribution. The ability to predict optimal features based on dataset characteristics opens doors for more efficient and effective machine learning model development.
Summary updated