TL;DR
This paper introduces a novel proximity-based, one-class classification method for authorship verification using Common N-Gram (CNG) dissimilarity. The method achieves competitive results on the PAN 2013 Author Identification benchmark, particularly excelling in confidence score ordering (AUC) and demonstrating strong performance on English and Spanish subsets.
What Problem Does This Paper Address?
The paper tackles the problem of authorship verification, which is a specific type of authorship attribution. Given a set of documents known to be written by a particular author, the task is to determine whether a new, unknown document was also written by that same author. This is distinct from traditional authorship attribution, where you choose the author from a predefined list. Authorship verification is crucial in various domains, including forensics, security, and literary analysis, where establishing the origin of a text is paramount. The challenge lies in developing robust methods that can accurately distinguish between texts by the same author and those by different authors, especially when dealing with diverse genres, languages, and potentially limited training data.
Methods
The core of the proposed method is a proximity-based one-class classification approach. Instead of building a model of what an author's writing is, it focuses on what it is not by measuring dissimilarity.
Here's a breakdown of the key techniques:
- Common N-Gram (CNG) Dissimilarity: This is the central measure for comparing documents. It quantifies the difference in frequencies of the most common character n-grams (sequences of characters) between two documents. The authors leverage a pre-existing CNG dissimilarity measure (Kešelj et al., 2003) but adapt its application.
- One-Class Classification Framework: The method operates within a one-class classification paradigm, meaning it only uses examples of the target author's writing during training. The goal is to classify whether a new document belongs to this "target class" (the author's writing).
- Proximity-Based Approach: The method is inspired by k-centers algorithms. For a given author, it considers all known documents as "centers." The dissimilarity between a new document and the author's known documents is then used for classification.
- Dissimilarity Ratio and Thresholding: A novel classification condition is proposed. For each known document $d_i$, the method calculates the maximum dissimilarity to any other known document ($D_{max}(d_i, A)$) and the dissimilarity to the sample document ($D(d_i, u)$). A dissimilarity ratio $r(d_i, u, A)$ is computed. The average of these ratios over all known documents, $M(u, A)$, is then compared to a learned threshold ($\theta$) to make the final decision. This ratio-based approach aims to capture how "outlier-like" the sample document is compared to the known documents.
- Ensembles of Classifiers: To improve robustness and mitigate parameter tuning issues, the authors explore ensembles of single one-class classifiers. These ensembles combine the decisions of multiple classifiers, each using different feature combinations (e.g., character vs. word n-grams, different profile lengths). They experiment with majority voting and weighted voting based on confidence scores.
Datasets
The primary dataset used for evaluation is the PAN 2013 Author Identification competition dataset. This dataset is multilingual, containing English, Greek, and Spanish subsets, and is specifically designed for authorship verification tasks.
To enrich their training data, the authors also compiled two additional datasets:
- mod-pan12-aa-EN: An English author verification set from the fiction corpus of the PAN 2012 competition.
- mod-Bpc-GR: A Greek author verification set from journal articles.
These additional datasets were used to train and tune some of their classifier variants, particularly for their final submission.
Key Insights and Ideas
- The power of "dissimilarity ratios" for one-class classification: Instead of just looking at raw dissimilarity, using a ratio that compares the sample document's dissimilarity to the most dissimilar known document provides a more nuanced measure of how much of an outlier the sample is.
- Ensembles are key to robustness: Combining multiple classifiers, each with different feature settings, significantly improves performance and reduces the need for extensive parameter tuning, especially for languages with limited training data.
- Character vs. Word N-grams: Both character and word n-grams have their strengths. Ensembles combining both types of features (e.g., eCW) generally yield the best results, suggesting that different levels of textual granularity capture complementary information for authorship verification.
- Greek subset challenges: The Greek subset of the PAN 2013 dataset proved particularly challenging for character-based CNG methods, likely due to specific dataset creation choices. This highlights the importance of considering language-specific characteristics and dataset construction.
- Confidence scores matter: The paper emphasizes the importance of confidence scores (measured by AUC) over binary accuracy, as real-world applications often require a measure of certainty. Their method shows strong performance in ordering instances by confidence.
Key Results
- Competitive Performance: The proposed method achieved competitive results on the PAN 2013 Author Identification task.
- AUC Excellence: Their final submission (vF1) ranked 1st out of 10 participants for AUC on the entire set and achieved the highest AUC on the English subset.
- Strong Accuracy: Ensembles combining character and word-based classifiers (eCW and eCW_sel) achieved the best accuracy on the entire set and the English subset, surpassing the best competition results. They also matched the best competition accuracy on the Spanish subset.
- Language Differences: Performance was consistently better on English and Spanish subsets compared to Greek.
- Ensemble Benefits: Weighted voting ensembles generally outperformed majority voting, especially when confidence scores were reliable.
Limitations and Future Work
- Parameter Tuning: While ensembles alleviate parameter tuning, single classifiers still require careful selection of parameters (n-gram length, profile length, threshold). The paper explores various tuning strategies.
- Greek Subset Performance: The method's performance on the Greek subset was lower than on English and Spanish, and generally outperformed by other participants. This is attributed to the specific nature of the Greek subset and its potential incompatibility with character-based CNG dissimilarity.
- Future Work: The authors suggest investigating the relationship between method performance and the number/length of texts. They also propose analyzing the role of word vs. character n-grams based on text genre and topical similarity.
Summary updated