Scaling Up Authorship Attribution

Jacob Striebel, Abishek R. Edikala, Ethan Irby, Alex Rosenfeld, J. Gage, Daniel Dakota, Sandra KüblerNAACL 2024 · DOI

TL;DR

This paper tackles the challenge of making authorship attribution systems more efficient and explainable, particularly for demanding government applications like the IARPA HIATUS program. They achieve significant speedups (up to 8.37x) in frame semantic parsing, a key component for generating interpretable linguistic features, with minimal impact on accuracy.

What Problem Does This Paper Address?

Authorship attribution, the task of identifying the author of a document, is crucial for various applications, including intelligence analysis and forensic linguistics. However, existing methods often struggle with real-world constraints like strict runtime limits and the need for explainability. The IARPA HIATUS program specifically aims to develop novel methods that are not only effective but also interpretable and deployable within these limitations. A major bottleneck identified is the computational cost of generating sophisticated linguistic features, particularly those derived from frame semantic parsing, which can be slow and resource-intensive. This paper addresses this by optimizing the frame semantic parsing pipeline to meet these stringent requirements.

Methods

The core of the paper's contribution lies in optimizing the generation of frame semantic features. Frame semantics, which analyzes the meaning of words and their relationships within a sentence, offers interpretable features that are harder for adversaries to evade compared to black-box neural models. To make this computationally expensive process feasible, the authors implemented a multi-processing architecture for their Frame Semantic Transformer (FST). This involves a controller-worker pattern where multiple GPUs are utilized to parallelize the parsing process.

They experimented with two versions of this architecture: "sentence groups subbatches" and "equal group subbatches." The latter, which loads the entire dataset into memory and distributes it, was chosen for its simplicity and efficiency. Additionally, they explored the impact of decoding beam size on parsing speed and accuracy. A smaller beam size generally leads to faster parsing but can degrade accuracy. The authors found a sweet spot by combining multiprocessing with a beam size of three, achieving a significant speedup with only a minor drop in accuracy.

For authorship attribution itself, they employ a multi-task learning architecture that synergizes neural and theory-driven features. This includes frame semantic features alongside other linguistic and stylistic features, aiming to create robust and explainable authorship embeddings.

Datasets

The authors used the Reddit Million User Dataset for development and experimentation. They selected a subset of authors and sentences from this dataset to reduce runtime for their experiments. For evaluating the frame semantic parser specifically, they used a test partition of FrameNet, consisting of 2,420 sentences.

Key Insights and Ideas

  • Frame semantics offers a powerful blend of interpretability and robustness: While complex, frame semantic features are valuable for authorship attribution because they are interpretable and difficult for adversaries to mimic.
  • Parallelization is key to unlocking complex features: The significant speedup achieved through multiprocessing demonstrates that computationally intensive feature extraction can be made practical for real-time applications.
  • Beam size is a tunable parameter for speed-accuracy trade-offs: The authors show that by carefully adjusting the beam size in frame semantic parsing, one can achieve substantial speed gains with minimal loss in accuracy, especially when combined with multiprocessing.
  • Authorship attribution can benefit from multi-task learning: By training an authorship encoder with auxiliary tasks (like frame semantic analysis), the system can learn more nuanced and distinguishable author embeddings.

Key Results

  • Frame Semantic Parser Speedup: The optimized system achieved a 2.5x increase in parsing speed by enabling multiprocessing across four GPUs. Further optimization by decreasing the beam size to three resulted in an almost 3x speed increase over the base parser, with negligible accuracy degradation.
  • Authorship Attribution Performance: The multiprocessing and beam search reduction yielded a system that took only 12% of the runtime of the original implementation for authorship attribution. The best performing configuration for authorship attribution was achieved with a beam size of three, which was almost 50% faster than the baseline.
  • Accuracy: The frame semantic parsing accuracy metrics (Trigger ID, Frame ID, Argument ID) showed only minor variations across different configurations, indicating that the speed optimizations did not significantly compromise the quality of the extracted features. Similarly, authorship attribution metrics (P@8 and MRR) showed only minimal differences.

Limitations and Future Work

The authors acknowledge that the longest document in their validation set is 350 words, which limits their ability to definitively test the system's performance on longer documents. They also note that while their system aims to identify malicious actors, it cannot guarantee zero false positives, and therefore, authorship attribution should be used as a supporting tool rather than a sole source of intelligence.

Future work suggested includes more robust parallelization of feature generation and automating the Docker image creation process for easier submission to evaluation infrastructures like HIATUS.

Summary updated

BibTeX
@inproceedings{striebel2024scaling,
  author = {Jacob Striebel and Abishek R. Edikala and Ethan Irby and Alex Rosenfeld and J. Gage and Daniel Dakota and Sandra Kübler},
  title = {{Scaling Up Authorship Attribution}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-industry.24},
}
Figure