Groningen Group E at SemEval-2024 Task 8: Detecting machine-generated texts through pre-trained language models augmented with explicit linguistic-stylistic features

Patrick Darwinkel, Sijbren Van Vaals, Marieke Van Der Holt, Jarno Van HoutenNAACL 2024

TL;DR

This paper tackles the growing challenge of detecting machine-generated text by combining powerful pre-trained language models (PLMs) with handcrafted linguistic-stylistic features. While their augmented models didn't outperform the official baselines, they demonstrated that these features offer valuable insights into the subtle differences between human and AI writing, particularly for multilingual tasks.

What Problem Does This Paper Address?

The rapid advancement and widespread adoption of Large Language Models (LLMs) like ChatGPT have made it increasingly difficult to distinguish between human-written and machine-generated text. This has significant implications for combating misinformation, academic integrity, and more. SemEval-2024 Task 8 specifically aims to address this by developing systems that can: (A) classify text as human or machine-generated, (B) identify the specific LLM that generated the text, and (C) pinpoint the exact boundaries within a text where machine generation begins. This paper focuses on exploring how linguistic and stylistic features, often overlooked by purely data-driven PLM approaches, can contribute to more robust and interpretable detection systems.

Methods

The core of the Groningen Group E's approach is to augment existing PLMs with a set of 20 linguistic-stylistic features. They experimented with two main PLM architectures:

  • Augmented DistilBERT: For subtasks A (human vs. machine) and B (generator identification), they fine-tuned DistilBERT (both English and multilingual versions) by adding a linear layer that processes the extracted linguistic features. This layer's output is then concatenated with DistilBERT's pooled output before the final classification layer.
  • Augmented Longformer: For subtask C (boundary detection), which requires token-level classification, they used an augmented Longformer. Here, the linguistic features are concatenated with the output state of each token, allowing the model to leverage contextual information about linguistic characteristics at a granular level.

They also explored a DistilBERT-embedded Random Forest approach, where DistilBERT embeddings (averaged using various pooling techniques) were concatenated with the linguistic features and fed into a Random Forest classifier. This served as a comparison to their end-to-end PLM augmentation.

The linguistic features themselves are diverse, covering:

  • Readability: Flesch-Kincaid, Coleman-Liau, Dale-Chall scores.
  • Entity Recognition: Ratios of entities to total words, and unique entities.
  • Syntax: Average parse tree height, noun phrase counts, and a syntactic complexity index (SYNNP).
  • Semantics: Lexical chain properties (number, length, span) and a controversy score.
  • Text Length Statistics: Average syllables per word and average sentence length.
  • Lexical Richness: Type-token ratio (TTR), Measure of Lexical Diversity in Text (MLTD), and hapax richness.

Datasets

The study utilized the datasets provided by the SemEval-2024 Task 8 organizers. These datasets consist of human-authored texts and machine-generated texts from 5 different LLMs (ChatGPT, Cohere, Davinci, Bloomz, Dolly). The data spans multiple domains and languages, including English, Chinese, Urdu, Russian, Indonesian, Arabic, and Bulgarian. The authors used the provided train and development sets for training and hyperparameter tuning, and the test set for final evaluation.

Key Insights and Ideas

  • Linguistic Features Offer Interpretability: While not always leading to the highest scores, the authors emphasize that linguistic-stylistic features provide a more interpretable understanding of why a text might be flagged as machine-generated, offering insights into specific LLM anomalies.
  • Multilingual Performance Boost: Contrary to expectations, their system performed better on the multilingual track for subtask A than the monolingual track, suggesting that some linguistic features might generalize well across languages or that the multilingual models are more robust.
  • Feature Importance Varies: Type-token ratio (TTR), MTLD, lexical chain properties, and average syllables per word were identified as particularly important features for distinguishing human from machine text.
  • PLMs Still Dominate: Despite the insights from handcrafted features, the augmented PLMs generally outperformed the feature-only Random Forest models, highlighting the continued power of large pre-trained models.
  • Domain Generalization Challenges: The system struggled with generalizing across different domains, indicating that LLM detection might be more domain-specific than initially assumed.

Key Results

  • Subtask A (Human vs. Machine):
    • Monolingual: Augmented DistilBERT achieved an accuracy of 0.63 (ranking 110th).
  • Multilingual: Augmented DistilBERT achieved an accuracy of 0.77 (ranking 41st).
  • Subtask B (Generator Identification): Augmented DistilBERT achieved an accuracy of 0.64.
  • Subtask C (Boundary Detection): Augmented Longformer achieved a Mean Absolute Error (MAE) of 26.07 (ranking 20th).

These results were generally lower than the task organizers' baselines (e.g., 0.88 for monolingual A, 0.75 for B, and 21.54 for C). However, their augmented DistilBERT model for subtask A monolingual slightly outperformed the organizer's baseline (0.75 vs. 0.74) on the development set.

Limitations and Future Work

  • Performance Gap: The authors acknowledge that their systems did not outperform the official baselines, indicating room for improvement.
  • Feature Generalization: The performance difference between monolingual and multilingual tasks suggests that feature extraction and applicability might need further refinement for diverse language sets.
  • Domain Specificity: The system's difficulty in generalizing across domains points to a need for more robust domain-adaptive techniques.
  • Future Research: The authors suggest that future work should focus on exploring new features, particularly for multilingual tasks, and investigating how to better combine feature-based approaches with PLMs for improved generalization and interpretability. They also note that the performance of feature-based methods might be better when the unseen test data is from the same distribution as the training data, suggesting that their proposed features do contain predictive power.

Summary updated

BibTeX
@inproceedings{darwinkel2024groningen,
  author = {Patrick Darwinkel and Sijbren Van Vaals and Marieke Van Der Holt and Jarno Van Houten},
  title = {{Groningen Group E at SemEval-2024 Task 8: Detecting machine-generated texts through pre-trained language models augmented with explicit linguistic-stylistic features}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure