Authorship Identification Using a Reduced Set of Linguistic Features

Stefan Ruseti, Traian RebedeaCLEF (Online Working Notes/Labs/Workshop) 2012 · OpenReview

TL;DR

This paper tackles authorship identification by proposing a surprisingly effective approach using a reduced set of linguistic features, challenging the common practice of using extensive feature sets. Their method achieves competitive results on the PAN 2012 competition by focusing on a few key features and a standard SVM classifier, demonstrating that simplicity can be powerful.

What Problem Does This Paper Address?

Authorship identification, the task of determining who wrote a piece of text, is a long-standing challenge in computer science. While the internet provides vast amounts of text, analyzing it for authorship is complicated by diverse writing styles and discourse types (e.g., emails vs. narratives). Previous research often relied on very large, complex feature sets, sometimes tailored specifically to certain text types. This paper aims to simplify this by investigating whether a smaller, more generalizable set of linguistic features can achieve comparable or even better results, particularly for short narrative texts, and whether the same approach can work for both "closed" (known authors) and "open" (unknown authors, including an "other" category) identification problems.

Methods

The authors' core contribution lies in their reduced set of linguistic features. Instead of a broad sweep, they focused on features identified as important in prior work but selected a subset that proved effective for the PAN 2012 corpus (short narratives). These include:

  • Character trigrams: Frequencies of three-character sequences.
  • POS bigrams and trigrams: Frequencies of adjacent part-of-speech tags (e.g., noun followed by verb). They noted that even simplified POS tags (like just the first letter) were informative.
  • Suffixes: Counting common English suffixes and their prevalence.
  • Word length: Analyzing the distribution of word lengths.
  • Syntactic complexity: Features derived from parsing sentences, such as average sentence length, tree depth, and dependency types.
  • Percentage of direct speech: To capture stylistic tendencies.

Crucially, they normalized these features to account for text length. For classification, they primarily used a Support Vector Machine (SVM) with the Sequential Minimal Optimization (SMO) algorithm, a standard and powerful technique. For the "open class" problem, they adapted the SVM output using a logistic regression model to estimate probabilities more accurately. They also experimented with Naive Bayes, finding it slightly less effective than SVM for this task.

Datasets

The primary dataset used was the PAN 2012 competition corpus, which consists of short narrative texts. The authors describe splitting the training texts into smaller pieces (5kB, 10kB, 20kB, 50kB) to create a larger number of training examples (100-200 per author) for better generalization, especially given the limited number of original training documents per author. The evaluation was performed on both the closed and open authorship identification tasks of this competition.

Key Insights and Ideas

  • Simplicity can be powerful: The most striking insight is that a reduced set of linguistic features, carefully chosen, can yield highly competitive results, challenging the assumption that more features are always better.
  • Generalizability across tasks: The same feature set and framework, with minor adjustments to the classification output, proved effective for both closed-set (identifying from a known group of authors) and open-set (identifying from a known group or classifying as "other") authorship identification.
  • Feature relevance for narratives: Features like character trigrams, POS tags, and syntactic complexity were found to be particularly useful for distinguishing authors in short narrative texts, where discourse-specific features (like those for emails) might be less relevant.
  • Effective data augmentation: Splitting training documents into smaller, sentence-aware chunks was a practical and effective way to overcome the challenge of having very few original training documents per author, enabling better classifier generalization.

Key Results

The paper reports strong performance on the PAN 2012 competition:

  • Closed-set identification (cross-validation): Achieved very high accuracy, with results like 100% for set A (using 5kB and 10kB slices) and 99.5% for set I (using 20kB and 50kB slices). While these are high, the authors acknowledge they are less relevant due to the similarity of training examples from the same document.
  • Closed-set identification (test corpus): The approach yielded good results on the actual test corpora, with accuracies ranging from 66.66% to 85.71% across different sets.
  • Overall performance: The solution achieved an overall document accuracy of 77%, ranking 3rd in the author attribution competition. The average accuracy over all 6 test sets was 76%, ranking 7th.

Limitations and Future Work

  • Limitations: The authors acknowledge that the results on the test corpus were lower than cross-validation, as expected. They also note that while their approach worked well, adding more "application-specific features" could potentially improve results further. The limited number of training documents per author was a significant constraint addressed by their splitting strategy.
  • Future Work: The paper suggests that further improvements could be made by incorporating more application-specific features. The success of their reduced feature set and splitting strategy implies that exploring similar simplified approaches for other text types or tasks could be fruitful. They also highlight the potential of logistic regression for estimating probabilities in open-set problems.

Summary updated

BibTeX
@inproceedings{ruseti2012authorship,
  author = {Stefan Ruseti and Traian Rebedea},
  title = {{Authorship Identification Using a Reduced Set of Linguistic Features}},
  booktitle = {CLEF (Online Working Notes/Labs/Workshop)},
  year = {2012},
  url = {https://openreview.net/forum?id=YrJfrjJM9H5},
}
Figure