Gated POS-Level Language Model for Authorship Verification

Linshu Ouyang, Yongzheng Zhang, Hui Liu, Yige Chen, Yipeng WangIJCAI 2020 · DOI

TL;DR

This paper introduces a novel "Gated POS-Level Language Model" for authorship verification. By leveraging Part-of-Speech (POS) tags to capture author-specific syntactic styles and employing a gated architecture for parameter sharing, the model achieves significantly improved accuracy, especially in cross-topic scenarios, while being more data-efficient than previous deep learning methods.

What Problem Does This Paper Address?

Authorship verification, the task of determining if a document was written by a specific author, has numerous applications in cybersecurity, plagiarism detection, and forensic analysis. Existing state-of-the-art methods often rely on character-level language models. However, these models struggle to capture crucial syntactic patterns, leading to suboptimal performance, particularly when the topics of the texts differ. Furthermore, deep learning models, while powerful, are notoriously data-hungry, and authorship verification datasets often have very limited text per author, making it difficult to train these models effectively without overfitting. This paper aims to bridge this gap by developing a more data-efficient and accurate authorship verification method that effectively captures author-specific syntactic styles.

Methods

The core innovation of this paper lies in its Gated POS-Level Language Model. Instead of directly processing characters or words, the authors first convert input texts into sequences of Part-of-Speech (POS) tags using a pre-trained POS tagger. This step is crucial because POS tags represent the grammatical structure of sentences, which is generally topic-agnostic and thus more indicative of an author's inherent writing style.

The POS sequences are then fed into a shared recurrent neural network (RNN) to capture contextual information. The novelty lies in the gated decoder architecture. This decoder has two main components:

  1. Shared Fully-Connected Layer: Learns common syntactic patterns shared across all authors.
  2. Author-Specific Fully-Connected Layer: Learns unique syntactic writing styles for each individual author.

A shared gate unit dynamically decides, at each step of the sequence, how much to rely on the common syntactic patterns versus the author-specific ones. This gated mechanism is designed to leverage the fact that most writing patterns are common, while only a small portion is author-specific. This approach helps prevent overfitting by making the decision based on the entire training set and makes the model more data-efficient by learning common patterns with shared parameters.

Datasets

The authors evaluated their model on four publicly available datasets commonly used in authorship verification research:

  • PAN-15: A dataset from a well-known authorship verification competition.
  • Gutenberg: An improved corpus derived from PAN-15, designed to mitigate biases and flaws. This dataset is highlighted as particularly important for fair evaluation.
  • Enron: Derived from public Enron emails, with manually selected and cleaned samples.
  • Reddit: Obtained from online discussions.

These datasets vary in genre and size, allowing for a comprehensive evaluation of the model's performance across different scenarios.

Key Insights and Ideas

  • Syntactic Style is Key for Cross-Topic Verification: The paper strongly argues that syntactic writing style, captured by POS tags, is more robust for authorship verification than character-level patterns, especially when texts have different topics. This is because grammatical structures are less dependent on the subject matter.
  • Data Efficiency Through Parameter Sharing and Gating: The gated architecture, which balances shared (common) and author-specific parameters, is crucial for effective learning with limited data. It allows the model to learn general linguistic rules efficiently while still specializing in individual author styles.
  • POS Tags Offer a Sweet Spot: While character-level models can overfit to specific character patterns (especially with noisy data like in PAN-15), and word-level models are too topic-dependent, POS tags provide a good balance of expressiveness and topic-agnosticism for authorship verification. The reduced vocabulary size of POS tags (around 19) also significantly reduces the model's parameter space.
  • Gating Prevents Overfitting: The gate mechanism, by learning from the entire dataset, helps to avoid overfitting to the limited training data of individual authors.

Key Results

The proposed Gated POS-Level Language Model consistently outperformed state-of-the-art competing methods across all four datasets.

  • Significant Accuracy Improvement: The model achieved notably higher AUC-ROC and AUC-PR scores compared to both traditional and deep learning baselines.
  • Cross-Topic Performance: The improvement was particularly pronounced in cross-topic scenarios, with gains of over 5% in AUC-ROC on the Gutenberg dataset.
  • Deep Learning Advantage: The results reaffirmed that deep learning methods, when properly applied, significantly outperform traditional feature-based approaches for authorship verification.
  • Effectiveness of POS and Gating: Ablation studies indicated that both the POS-level input and the gated architecture contributed significantly to the performance gains. The gated model outperformed a multi-head architecture when both used POS inputs.

Limitations and Future Work

  • POS Tagger Choice: The paper acknowledges that the choice of POS tagger can influence results, though they arbitrarily chose a popular open-source one (Spacy). They suggest that other taggers could be explored.
  • RNN vs. LSTM: While they used a vanilla RNN for simplicity and found it sufficient, they speculate that LSTMs might offer limited improvements due to data scarcity preventing them from learning meaningful long-range correlations.
  • Future Work: The authors suggest exploring more explicit syntactic information, such as dependency parsing, although their initial experiments with it were less successful than POS tags. Further research could also investigate how to best adapt these models to even shorter texts or different types of stylistic variations.

Summary updated

BibTeX
@inproceedings{ouyang2020gated,
  author = {Linshu Ouyang and Yongzheng Zhang and Hui Liu and Yige Chen and Yipeng Wang},
  title = {{Gated POS-Level Language Model for Authorship Verification}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.24963/ijcai.2020/557},
}
Figure