PetKaz at SemEval-2024 Task 8: Can Linguistics Capture the Specifics of LLM-generated Text?

Kseniia Petukhova, Roman Kazakov, Ekaterina KochmarNAACL 2024

TL;DR

This paper tackles the challenge of detecting text generated by Large Language Models (LLMs), a crucial task for mitigating misuse. The authors propose a black-box approach combining LLM embeddings with linguistic features, demonstrating that a carefully selected training set and diverse linguistic features can achieve high accuracy, even outperforming a baseline model.

What Problem Does This Paper Address?

The rapid advancement of Large Language Models (LLMs) has led to their ability to generate text that is often indistinguishable from human-written content. This poses a significant challenge for detecting machine-generated text (MGT), which is essential for combating misinformation, academic dishonesty, and other potential misuses. Existing MGT detection methods can be broadly categorized into black-box (relying on API access and extracted features) and white-box (requiring model internals). This paper focuses on the black-box approach, aiming to develop a robust detector that can generalize across various LLMs and domains without needing access to the underlying models.

Methods

The PetKaz submission employs a black-box MGT detection system built upon three core components:

  1. Embeddings: They fine-tune a RoBERTa-base autoencoder model on the HWT vs. MGT classification task and then use the [CLS] token embeddings as input features.
  2. Linguistic Features: A diverse set of linguistically motivated features are extracted, including:
  3. Text Statistics: Number of difficult words, lexicon count, and sentence count.
  4. Readability: Flesch Reading Ease, Flesch-Kincaid Grade Level, and Linsear Write Metric.
  5. Stylometry: Based on unigrams, bigrams, and "spelling signatures," with dimensionality reduced via SVD.
  6. Lexical Diversity: Utilizing measures like Type-Token Ratio (TTR) and Maas TTR.
  7. Rhetorical Structure Theory (RST): Counting occurrences of various rhetorical relations.
  8. Entity Grid: Analyzing patterns of entity transitions across sentences.
  9. Feed-Forward Neural Network: A concatenation of the extracted embeddings and linguistic features is fed into a feed-forward neural network for binary classification (HWT vs. MGT).

A key aspect of their methodology is the investigation into the impact of training data selection, particularly focusing on the WikiHow domain for its perceived higher consistency in LLM-generated texts.

Datasets

The primary dataset used is an extension of the M4 dataset (Wang et al., 2023). This dataset encompasses various domains (WikiHow, Wikipedia, Reddit, arXiv, PeerRead, and Outfox) and texts generated by multiple LLMs (ChatGPT, Cohere, Davinci003, Dolly-v2, BLOOMZ, and GPT-4), alongside human-written texts (HWTs). The organizers provided balanced training, development, and test sets. The test set was specifically limited to the Outfox domain.

Key Insights and Ideas

  • Linguistics Matter: The paper strongly suggests that linguistic features, even when used without LLM embeddings, can capture significant differences between human and machine-generated text.
  • Training Data is Crucial: The choice of training data significantly impacts performance. Specifically, using HWTs only from the WikiHow domain, which exhibits higher consistency in LLM outputs, led to improved performance when combined with embeddings.
  • Diversity as a Complement: Lexical diversity features, when combined with embeddings, proved to be a particularly effective combination, boosting accuracy beyond what embeddings alone could achieve. This suggests diversity features complement embedding representations well.
  • Stylometry's Strength: Stylometry features, when used in isolation, performed better than other linguistic features, indicating their ability to capture general text patterns.
  • RST's Limited Impact: Rhetorical Structure Theory features did not appear to significantly aid in distinguishing MGTs from HWTs, suggesting that the hierarchical organization of text might be similar across human and machine generation in this context.

Key Results

  • The best-performing model, using a combination of embeddings and lexical diversity features trained on a reduced WikiHow-centric training set, achieved an accuracy of 0.95 on the development set.
  • On the test set, their main submitted configuration (embeddings + diversity features on the reduced set) achieved an accuracy of 0.91 and an F1 score of 0.92, ranking 12th out of 124 teams in the SemEval-2024 Task 8.
  • Their model outperformed the organizers' baseline (accuracy 0.88) on the test set.
  • The model demonstrated generalizability, performing well on previously unseen models (GPT-4, BLOOMZ) and across domains, even though the test set was limited to Outfox.
  • The classifier achieved near-perfect precision on texts from GPT-4 and Davinci003, with BLOOMZ being the most challenging model (8% misclassification rate).
  • Approximately 18% of HWTs were incorrectly classified as machine-generated, highlighting the difficulty in achieving perfect separation.

Limitations and Future Work

  • English Language Only: The work is limited to the English language due to participation in a single subtask.
  • Domain and LLM Specificity: The generalizability of the approach beyond the specific domains and LLMs included in the shared task data needs further verification.
  • Future Research: The authors suggest future work could involve in-depth analysis of training data selection techniques and expanding the set of linguistic features. They also highlight the potential for linguistic features (stylometry, entity grid) to perform competitively even without LLM embeddings, and the benefit of careful training data selection for embedding-based models.

Summary updated

BibTeX
@inproceedings{petukhova2024petkaz,
  author = {Kseniia Petukhova and Roman Kazakov and Ekaterina Kochmar},
  title = {{PetKaz at SemEval-2024 Task 8: Can Linguistics Capture the Specifics of LLM-generated Text?}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure