Style Change Detection Based On Bert And Conv1d

Qidi Lao, Li Ma, Wenyin Yang, Zexian Yang, Dong Yuan, Zhenlin Tan, Langzhang LiangCLEF (Working Notes) 2022 · OpenReview

Introduction

Identifying authorship shifts within a single document is a critical task in digital forensics and plagiarism detection. Known as Style Change Detection, this challenge involves determining if a document was authored by one person or stitched together from multiple sources. This paper presents a hybrid deep learning approach that combines the contextual power of BERT with the local feature extraction capabilities of 1D Convolutional Neural Networks (Conv1D) to detect author switches at both paragraph and sentence levels.

Problem Definition

The authors address three specific challenges defined by the PAN 2022 Evaluation Laboratory, varying in complexity and granularity:

  1. Style Change Basic (Task 1): Detecting a single authorship switch between two authors at the paragraph level.
  2. Style Change Advanced (Task 2): Identifying multiple switches among two or more authors and assigning each paragraph to the correct author.
  3. Style Change Real-World (Task 3): Detecting authorship changes at the sentence level, which is significantly more difficult due to the limited text available for stylistic analysis.
Visualization of the three style change detection tasks across example documents.

The core difficulty lies in distinguishing between different writing styles on the same topic without being misled by the semantic content itself.

Methodology: BERT + Conv1D Architecture

The proposed pipeline treats style change detection as a binary classification task: given two adjacent segments of text, the model predicts whether they were written by the same author (0) or different authors (1).

The BERT+Conv1D model pipeline for binary style change classification.

The architecture consists of several key components:

  • BERT (Bidirectional Encoder Representations from Transformers): A pre-trained 12-layer BERT model processes the input text to capture deep, bidirectional context and semantic nuances.
  • Conv1D (One-Dimensional Convolution): A Conv1D layer is applied to the BERT output to extract local stylistic features. This acts as a sliding window searching for specific "signatures" in word vectors that characterize an author's style.
  • Max Pooling & Classification: Salient features from the convolution are identified via max-pooling and passed to a fully connected layer for final classification.

For Task 2, which requires multi-author attribution, the model performs pairwise comparisons between paragraphs to cluster them by similarity.

Experimental Setup and Datasets

The study utilized the PAN 2022 competition datasets, consisting of English documents with up to five different authors. The data was split into training (70%), validation (15%), and testing (15%) sets. The datasets provided ground truth labels for both paragraph-level (Tasks 1 & 2) and sentence-level (Task 3) switches.

Results and Discussion

The model's performance was evaluated using the F1 score on the hidden test set.

Task Description Test F1 Score
Task 1 Paragraph level, 2 authors 0.7471
Task 2 Paragraph level, 2+ authors + attribution 0.4170
Task 3 Sentence level, 2+ authors 0.6314

Key Insights

  • Style as Similarity: Framing document segmentation as a series of local similarity checks proved effective. A style change is flagged when the similarity between adjacent segments drops below a threshold.
  • Sentence-Level Robustness: Despite the reduced data per segment in Task 3, the BERT+Conv1D approach remained robust, achieving a respectable 0.6314 F1 score.
  • Error Propagation in Attribution: The lower score in Task 2 highlights a "domino effect" where a single incorrect prediction in a sequence of comparisons can lead to the mislabeling of all subsequent paragraphs.

Conclusion and Future Work

While the BERT+Conv1D hybrid is effective at identifying where changes occur, multi-author attribution remains a significant challenge. Future work could involve moving beyond sequential comparisons to a more global strategy for author assignment to mitigate error propagation. Additionally, utilizing larger architectures like RoBERTa or more modern large-scale pre-trained models could capture even finer stylistic nuances, potentially overcoming hardware-related constraints encountered in this study.

Summary updated

BibTeX
@inproceedings{lao2022style,
  author = {Qidi Lao and Li Ma and Wenyin Yang and Zexian Yang and Dong Yuan and Zhenlin Tan and Langzhang Liang},
  title = {{Style Change Detection Based On Bert And Conv1d}},
  booktitle = {CLEF (Working Notes)},
  year = {2022},
  url = {https://openreview.net/forum?id=4efACQc2Eb},
}
Figure