Sub-Profiling by Linguistic Dimensions to Solve the Authorship Attribution Task

Upendra Sapkota, Thamar SolorioClef (online Working Notes/labs/workshop) 2012

TL;DR

This paper introduces a novel approach to authorship attribution by breaking down an author's writing style into distinct "linguistic modalities" (e.g., stylistic, syntactic, semantic). By creating separate "sub-profiles" for each modality and then combining their predictions, the authors achieve competitive results on the PAN 2012 challenge, particularly for the challenging closed-class task.

What Problem Does This Paper Address?

Authorship attribution (AA) is the task of identifying the author of an anonymous text. This has applications in areas like plagiarism detection and cybersecurity. AA tasks typically fall into two categories: closed-class (where the author is guaranteed to be among a known set of candidates) and open-class (where the author might be unknown). The PAN 2012 challenge presented a particularly difficult scenario with limited training data per author, making traditional machine learning approaches challenging. Existing profile-based methods often create a single profile per author, which might not capture the nuances of their writing style across different linguistic dimensions.

Methods

The authors propose a modified profile-based approach that leverages the concept of "linguistic modalities." Instead of a single author profile, they create multiple "sub-profiles" for each author, with each sub-profile representing a specific linguistic dimension. These modalities include:

  • Stylistic: Features like sentence length, punctuation usage, and contraction frequency.
  • Syntactic: Part-of-speech tag unigrams, bigrams, trigrams, and grammatical relations.
  • Semantic: Top words after stop-word removal.
  • Character n-grams: Character trigrams.
  • Word n-grams: Word unigrams, bigrams, and trigrams.

For each modality, they concatenate all training documents from an author to build their sub-profile. When a test document arrives, its similarity to each author's sub-profile is computed using cosine similarity. Each modality then makes an independent prediction for the author. Finally, these individual modality predictions are combined using a majority voting mechanism to arrive at the final classification. For the open-class problem, they introduce a threshold-based mechanism to identify "unknown" authors when confidence scores are low.

Datasets

The authors evaluated their system on datasets from the PAN 2012 authorship attribution competition. These datasets included both closed-class and open-class tasks, with varying numbers of authors (3, 8, and 14). The training data was consistent across closed and open-class problems for each author set, while the test data differed.

Key Insights and Ideas

  • Decomposition of Writing Style: The core idea of breaking down an author's "writeprint" into distinct linguistic modalities is powerful. This allows for a more granular analysis of writing style, potentially capturing different aspects of an author's behavior that might be masked in a single, aggregated profile.
  • Modality-Specific Voting: Treating each modality as an independent predictor and then combining their votes (majority voting) is a robust ensemble strategy. It leverages the strengths of different feature sets and mitigates the weaknesses of any single modality.
  • Handling Limited Data: The sub-profiling approach, combined with profile-based methods, proves effective in scenarios with very limited training data, a key challenge in the PAN 2012 competition.
  • Open-Class "Unknown" Detection: The proposed threshold mechanism for identifying "unknown" authors in the open-class setting is a practical approach to handle cases where a test document doesn't clearly belong to any of the candidate authors.

Key Results

  • Closed-Class Performance: The system achieved highly competitive results on the closed-class AA task, matching the highest accuracies reported in the competition for several datasets. For example, on the 3-Author dataset, syntactic and character n-gram modalities achieved 100% accuracy, and the majority voting combined result was also 100%. On the 8-Author dataset, the majority voting achieved 100% accuracy.
  • Open-Class Performance: While not as universally dominant as in the closed-class task, the system still performed competitively on the open-class problem. For the 8-Author dataset, their system achieved the highest accuracy (76.47%) among participants. The majority voting combination was not always the best performer, suggesting potential issues with weak predictors in the ensemble for some datasets.

Limitations and Future Work

  • Open-Class Challenges: The authors acknowledge that their open-class performance, particularly for the 14-author dataset, was not impressive. They suggest that further investigation is needed to find better ways to handle open-class settings.
  • Parameter Tuning: While they used a validation set for parameter tuning (e.g., feature selection percentages), the paper implies that further empirical analysis could explore different parameter values, especially for the "unknown" detection threshold in the open-class setting.
  • Dataset Specificity: The performance on the 14-author open-class dataset indicates that the algorithm's effectiveness might be dataset-dependent, and more robust generalization is desirable.

Summary updated

BibTeX
@inproceedings{sapkota2012sub,
  author = {Upendra Sapkota and Thamar Solorio},
  title = {{Sub-Profiling by Linguistic Dimensions to Solve the Authorship Attribution Task}},
  booktitle = {Clef (online Working Notes/labs/workshop)},
  year = {2012},
}
Figure