TL;DR
This paper introduces GAN-BERT, a novel approach for authorship attribution in late 19th-century novels, addressing the challenge of limited data per author. By combining Generative Adversarial Networks (GANs) with BERT, they achieve strong performance, demonstrating that transfer learning and careful data sampling are crucial for effective attribution in this domain.
What Problem Does This Paper Address?
Authorship attribution, the task of identifying the author of a text, is particularly challenging and important in literary studies. Historically, authors, especially women, used pen names, leading to misattributions. While deep learning models have shown promise, existing methods often struggle with imbalanced datasets and limited text per author, making reproducibility and generalization difficult. This paper focuses on late 19th-century novels, a period rich with literary works but also presenting challenges due to the nature of authorship and data availability. The authors aim to develop a robust authorship attribution model that can handle these real-world complexities, specifically by exploring the impact of data sampling strategies and transfer learning.
Methods
The core of this research is the GAN-BERT model, an adaptation of a previously proposed GAN-BERT architecture. Unlike the original, which used semi-supervised learning with labeled and unlabeled data, this work trains the GAN-BERT model using only labeled data. The model comprises a generator (G) and a discriminator (D). The generator creates synthetic "fake" text samples, while the discriminator, enhanced by BERT embeddings, learns to distinguish between real and fake texts and, crucially, to classify texts into one of N+1 classes (N authors plus a "fake" class). This adversarial training helps the model learn robust representations that are less susceptible to simple stylistic mimicry.
A key methodological contribution is the extensive experimentation with data sampling strategies. The authors investigate the impact of:
- Random Sampling vs. Manual Selection: Comparing performance when authors are manually grouped versus when combinations are randomly sampled.
- Transfer Learning: Training a base model on a larger, 20-author dataset and then fine-tuning it on smaller subsets of authors. This is crucial for scenarios with limited data per author.
- Number of Authors: Systematically varying the number of authors in the training/testing sets to understand its effect on performance.
- Text Sample Size per Novel: Analyzing how the amount of text extracted from each novel influences attribution accuracy.
Datasets
The primary dataset used in this research is a newly curated dataset of late 19th-century English novels. This dataset was derived from Project Gutenberg and underwent significant manual validation to ensure quality. Novels were filtered to have a word count greater than 10,000. The initial master dataset contained 1232 novels from 62 authors, segmented by publication period. For the core experiments, a subset focusing on the late 19th century (1871-1900) was selected, comprising 541 novels. From this, the authors narrowed down to the top 20 authors with the most novels, creating a balanced dataset of 400 novels by 20 authors for training and evaluation.
For baseline comparisons, the authors also used subsets of existing datasets:
- IMDB20: A subset of 20 authors from the IMDb dataset.
- Blog20: A subset of 20 authors from the Blog Authorship dataset.
Key Insights and Ideas
- GAN-BERT's Robustness: The GAN-BERT architecture, by generating synthetic data, helps the model learn to distinguish genuine authorship even when faced with potential stylistic obfuscation or forgery.
- Transfer Learning is Key: For authorship attribution with limited data per author, transfer learning from a larger, more general model significantly boosts performance, especially when dealing with fewer authors.
- Random Sampling Outperforms Manual Selection: Surprisingly, random sampling of author combinations, when done at scale (50-100 combinations), yielded better and more consistent results than manually curated sample sets, suggesting that manual selection can introduce biases.
- Diminishing Returns with More Authors: As the number of authors in the dataset increases, the model's performance generally declines, highlighting the inherent difficulty of distinguishing between a larger pool of authors.
- Sample Size Matters, But Not Always More is Better: While a minimum text sample size is necessary for effective attribution, increasing the sample size beyond a certain point (e.g., 5-10 chunks of 512 words) can negatively impact performance, potentially due to increased data variance or overfitting.
Key Results
- The proposed GAN-BERT model achieved accuracy exceeding 0.88 in some configurations.
- Random sampling of author combinations (50 and 100 samples) achieved higher accuracy than manually selected sets, often exceeding 0.97%.
- Transfer learning significantly improved performance, with the best results observed for the 2-author case and a decline as the number of authors increased.
- For the 18-author classification, increasing text sample size from 5 to 35 chunks per novel showed a negative impact on performance, with accuracy dropping from 0.92+ to 0.86.
- In baseline experiments, the proposed GAN-BERT model outperformed stylometric and character n-gram models but was slightly outperformed by TF-IDF and BertAA on specific datasets.
Limitations and Future Work
The authors acknowledge several limitations:
- Limited Author Scope: The study focused on a limited number of authors from the late 19th century, which might limit the generalizability of the findings. Future work should consider the entire dataset of long 19th-century novelists.
- Copyright Restrictions: Due to copyright, the full dataset is not released, only scripts for reproduction.
- Feature Set: The current model could potentially benefit from incorporating richer feature sets, such as TF-IDF or more advanced stylometric features, to enhance performance on specific datasets.
Promising future directions include:
- Exploring few-shot and zero-shot learning scenarios.
- Experimenting with different GAN and transformer model architectures.
- Investigating the impact of cross-domain authorship attribution.
Summary updated