Datasets and Models for Authorship Attribution on Italian Personal Writings

Gaetana Ruggiero, Albert Gatt, Malvina Nissim

Abstract

Existing research on Authorship Attribution (AA) focuses on texts for which a lot of data is available (e.g., novels), mainly in English. We approach AA via Authorship Verification on short Italian texts in two novel datasets, and analyze the interaction between genre, topic, gender, and length. Results show that AV is feasible even with little data, but more evidence helps. Gender and topic can be indicative clues, and if not controlled for, they might overtake more specific aspects of personal style.

Introduction

Authorship Attribution (AA)—the task of identifying an author based on writing style—has historically focused on long-form English texts, such as novels or news articles. However, real-world applications like detecting plagiarism or identifying malicious online accounts often involve short, informal snippets. Italian AA research has been largely limited to high-profile literary mysteries, leaving a significant gap in resources for "low-resource" scenarios: short, user-generated Italian content where personal style might be masked by informal grammar or specific topics. This paper introduces two novel datasets, ForumFree and Diaries, and demonstrates that Authorship Verification (AV) is highly effective even for short texts when variables like topic and gender are carefully controlled.

Methodology

We employed the GLAD system, a well-regarded framework for authorship verification that utilizes a Support Vector Machine (SVM). GLAD extracts 24 different features, including character-level n-grams, stylometry, entropy, and data compression measures. To ensure the model learned individual style rather than broad demographic clusters, we implemented several preprocessing techniques:

  • NER Masking: We used spaCy to replace specific names, locations, and organizations with generic labels (e.g., PER, LOC, ORG) to prevent the model from using explicit clues.
  • Text Bleaching: Tokens were transformed into abstract representations (e.g., "House" becomes "ULLLL") to mask lexical content while preserving structural features like word length and punctuation.
  • Controlled Experiments: We designed intra-topic, cross-topic, and cross-genre tests, specifically controlling for gender and the amount of text (evidence) available per author.
Construction of Authorship Verification problems from author texts.

As illustrated in the figure above, raw texts from an author are partitioned into "known" and "unknown" sets to create the binary classification problems required for the AV task.

Datasets

The study introduces two new benchmarks for the Italian language:

  1. ForumFree: A collection of web forum comments covering two main topics: "Aesthetic Medicine" and "TV Programmes/Celebrities."
  2. Diaries: A dataset of personal diary fragments from the project Italiani all’estero (Italians abroad).

These datasets are reformatted into AV "problems," where the system is presented with a "known" text and an "unknown" text of equal length to compare.

Results and Discussion

Our experiments revealed that Authorship Verification is remarkably effective for Italian personal writings, provided sufficient evidence is available.

The Impact of Evidence Quantity

There is a clear correlation between the amount of text available and accuracy. Performance improved significantly as the "evidence" increased from 400 to 3,000 words per author.

Evaluation results on the ForumFree dataset across different word counts per author.

As shown in the table above, the ForumFree (Mixed Topics) subset achieved a near-perfect AUC of 1.000 and a c@1 score of 0.966 when 3,000 words of evidence were available. Despite the rise of deep learning, SVM-based models like GLAD remain state-of-the-art for AV in low-resource scenarios where large-scale Transformer models tend to overfit.

Topic, Gender, and Bleaching

  • The "Topic and Gender" Trap: If gender and topic aren't controlled, the classifier might perform well simply by clustering authors into broad groups rather than identifying unique stylistic fingerprints.
  • Bleaching Resilience: For the highly personal Diaries dataset, "bleaching" the text actually improved scores, suggesting that lexical content in diaries can sometimes act as noise rather than a signal for authorship.
  • Cross-Genre Performance: Training on forums and testing on diaries (or vice versa) yielded respectable results (around 0.74 AUC), proving that some stylistic markers transcend specific platforms.

Conclusion

This research demonstrates that AV is feasible for short Italian texts, with performance scaling strongly with text length. While 3,000 words yielded the best results, many real-world forensic cases involve much less data. Future work is needed to maintain high accuracy with only a few hundred words and to explore gender as a more nuanced social construct in linguistic models.

Summary updated

Abstract

Existing research on Authorship Attribution (AA) focuses on texts for which a lot of data is available (e.g novels), mainly in English. We approach AA via Authorship Verification on short Italian texts in two novel datasets, and analyze the interaction between genre, topic, gender and length. Results show that AV is feasible even with little data, but more evidence helps. Gender and topic can be indicative clues, and if not controlled for, they might overtake more specific aspects of personal style.
BibTeX
@inproceedings{ruggiero2020datasets,
  author = {Gaetana Ruggiero and Albert Gatt and Malvina Nissim},
  title = {{Datasets and Models for Authorship Attribution on Italian Personal Writings}},
  booktitle = {CLiC-it},
  year = {2020},
  url = {https://openreview.net/forum?id=I3UT8zdbti_},
}
Figure