Syntactic Recurrent Neural Network for Authorship Attribution

Fereshteh Jafariakinabad, Sansiri Tarnpradab, Kien A. HuaarXiv 2019 · OpenReview

TL;DR

This paper introduces a novel Syntactic Recurrent Neural Network (SRNN) for authorship attribution that focuses on grammatical structure rather than word choice. By encoding sentence syntax using Part-of-Speech (POS) tags, their model achieves a significant 14% accuracy improvement over lexical-based methods on the PAN 2012 dataset, demonstrating the robustness of syntactic features against varying topics.

What Problem Does This Paper Address?

Authorship attribution, the task of identifying the author of a text, has traditionally relied on analyzing an author's "writing style." While lexical features (word choice) are intuitive and have been widely explored, they suffer from a significant drawback: they are heavily influenced by the text's content or topic. This makes lexical models less effective when dealing with diverse datasets where authors might write on many different subjects. The paper addresses this by proposing a content-independent approach, focusing on syntactic features (how sentences are constructed) which are more stable across different topics and genres, thus offering a more robust solution for authorship attribution.

Methods

The core of the proposed method is a Syntactic Recurrent Neural Network (SRNN) designed to capture hierarchical syntactic patterns. The process involves several key steps:

  1. POS Embedding: Each word in a sentence is converted into its Part-of-Speech (POS) tag (e.g., noun, verb, adjective). These tags are then embedded into low-dimensional vectors. This is advantageous over word embeddings as the vocabulary of POS tags is fixed and much smaller, avoiding out-of-vocabulary issues.
  2. POS Encoder: This layer learns the syntactic representation of each sentence. The authors explore two approaches:
  3. Convolutional Neural Networks (CNNs): Used to capture short-term dependencies between POS tags, effectively identifying local syntactic patterns (like n-grams).
  4. Long Short-Term Memory (LSTM) networks: Used to capture long-term dependencies within a sentence's POS tag sequence, understanding more complex grammatical structures.
  5. The paper also investigates using bidirectional LSTMs for sentence encoding.
  6. Sentence Encoder: A bidirectional LSTM aggregates the sentence-level syntactic representations into a document-level representation. This captures how sentences are structured together within a document.
  7. Attention Mechanism: To further refine the document representation, an attention mechanism is employed. This allows the model to weigh sentences differently based on their importance in revealing the author's style, focusing on the most informative parts of the text.
  8. Classification: Finally, a softmax classifier uses the attention-weighted document representation to predict the author.

Datasets

The primary dataset used for evaluation is the PAN 2012 authorship attribution dataset. Specifically, the authors used Task I of this dataset, which involves authorship attribution among a closed set of 14 authors. The dataset consists of novel-length documents, with training sets containing two documents per author and test sets containing one document per author. The authors mention that for training, novels were segmented into sequences of sentences to create enough training samples.

Key Insights and Ideas

  • Syntax Trumps Lexicon for Robustness: The most significant insight is that syntactic features, being content-independent, are more robust for authorship attribution, especially in diverse topic environments, compared to lexical features.
  • Hierarchical Encoding is Key: The proposed hierarchical structure, encoding POS tags into sentence representations and then aggregating these into document representations, effectively captures complex stylistic nuances.
  • LSTM vs. CNN for Syntax: While both CNNs and LSTMs are explored for POS encoding, LSTMs (particularly bidirectional ones) seem to be more adept at capturing the longer-range syntactic dependencies crucial for style.
  • Attention Improves Focus: The attention mechanism helps the model prioritize sentences that are more indicative of authorial style, leading to better overall performance.
  • Short Documents are Harder: The study suggests that identifying authorial style in shorter documents is more challenging, as confirmed by performance drops when using fewer sentences per segment.

Key Results

  • The proposed Syntactic Recurrent Neural Network (SRNN) achieved a 14% improvement in segment-level accuracy and a 7.15% improvement in document-level accuracy compared to the best lexical baseline on the PAN 2012 dataset.
  • The syntactic models (both CNN-LSTM and LSTM-LSTM) achieved 100% document-level accuracy, correctly classifying all 14 novels in the test set. This significantly outperformed the best lexical model, which achieved 85.71% document-level accuracy.
  • Syntactic CNN-LSTM slightly outperformed syntactic LSTM-LSTM in segment-level accuracy (78.76% vs. 74.40%), but LSTM-LSTM models were observed to be quicker in capturing authorial style, making them potentially better for short documents.
  • The study found that increasing the number of convolutional layers in the CNN POS encoder generally decreased performance, likely due to overfitting with limited data.

Limitations and Future Work

  • Data Limitation: The authors acknowledge that performance on the test set dropped significantly, attributing this to insufficient training data. They suggest that with more writing samples per author, test results would be closer to validation results.
  • Short Documents: While LSTMs showed promise for short documents, the general challenge of style detection in short texts remains an area for further exploration.
  • Future Work: The paper implicitly suggests exploring more advanced hierarchical models, different POS tag sets, and potentially incorporating structural features alongside syntactic ones for even more comprehensive authorship attribution. The authors also note that LSTM-based POS encoders are quicker at capturing style, suggesting their suitability for scenarios with limited text.

Summary updated

Abstract

Writing style is a combination of consistent decisions at different levels of language production including lexical, syntactic, and structural associated to a specific author (or author groups). While lexical-based models have been widely explored in style-based text classification, relying on content makes the model less scalable when dealing with heterogeneous data comprised of various topics. On the other hand, syntactic models which are content-independent, are more robust against topic variance. In this paper, we introduce a syntactic recurrent neural network to encode the syntactic patterns of a document in a hierarchical structure. The model first learns the syntactic representation of sentences from the sequence of part-of-speech tags. For this purpose, we exploit both convolutional filters and long short-term memories to investigate the short-term and long-term dependencies of part-of-speech tags in the sentences. Subsequently, the syntactic representations of sentences are aggregated into document representation using recurrent neural networks. Our experimental results on PAN 2012 dataset for authorship attribution task shows that syntactic recurrent neural network outperforms the lexical model with the identical architecture by approximately 14% in terms of accuracy.
BibTeX
@misc{jafariakinabad2019syntactic,
  author = {Fereshteh Jafariakinabad and Sansiri Tarnpradab and Kien A. Hua},
  title = {{Syntactic Recurrent Neural Network for Authorship Attribution}},
  year = {2019},
  eprint = {1902.09723},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1902.09723},
}
Figure