RFBES at SemEval-2024 Task 8: Investigating Syntactic and Semantic Features for Distinguishing AI-Generated and Human-Written Texts

Mohammad Heydari Rad, Farhan Farsi, Shayan Bali, Romina Etezadi, Mehrnoush ShamsfardNAACL 2024

TL;DR

This paper tackles the growing challenge of distinguishing AI-generated text from human-written text by exploring both semantic and syntactic features. They developed models that achieved high accuracy on the M4 dataset, finding that semantic analysis, particularly when applied to smaller text segments, was more effective than syntactic analysis alone.

What Problem Does This Paper Address?

The rapid advancement and accessibility of Large Language Models (LLMs) like ChatGPT and Gemini have made it increasingly difficult to differentiate between AI-generated and human-written content. This poses significant concerns across various domains, including the spread of misinformation, academic integrity, and content moderation. While many existing approaches focus on semantic cues, this paper aims to provide a more comprehensive solution by investigating the utility of both semantic and syntactic features for this classification task.

Methods

The authors propose a two-pronged approach:

  • Semantic Approach: They employed a fine-tuned XLM-ROBERTa model. A key innovation here is their text segmentation strategy, where longer texts are broken down into smaller units. This allows for a more granular analysis of semantic nuances that might be missed in a holistic view. They then explored three methods to combine the segment-level predictions: soft voting, hard voting, and weighted soft voting (where segment weight is based on word count).
  • Syntactic Approach: For syntactic analysis, they utilized Part-of-Speech (POS) tags generated by Trankit. They built a stacked, bidirectional LSTM model with an attention mechanism to capture syntactic patterns. The rationale behind using LSTMs was to avoid the potential overshadowing of syntactic features by semantic meaning inherent in transformer models.

Datasets

The primary dataset used for this research is the M4 dataset (Wang et al., 2023). This dataset is designed for the SemEval-2024 Task 8 and includes both multilingual and monolingual tracks.

  • Multilingual Track: Encompasses human-written and AI-generated texts from six languages: English, Arabic, Chinese, Indonesian, Russian, and Urdu.
  • Monolingual Track: Specifically uses the English portion of the dataset.

The M4 dataset contains AI-generated texts from various LLMs (e.g., ChatGPT, text-davinci-003, LLaMa) and covers diverse tasks like generating Wikipedia articles, peer reviews, and news briefs.

Key Insights and Ideas

  • Segmentation is Key for Semantics: Analyzing text in smaller segments, rather than as a whole, significantly improves the effectiveness of semantic-based AI-generated text detection. This suggests that AI models might exhibit more consistent semantic patterns within these smaller chunks.
  • Semantic Dominance: The semantic approach, especially with weighted soft voting, outperformed the syntactic approach. This indicates that current AI generation methods leave more discernible semantic footprints than syntactic ones.
  • Syntactic Analysis Limitations: The syntactic approach, using POS tags with an LSTM, showed no significant difference between AI-generated and human-written texts. This suggests that current LLMs are adept at mimicking human syntactic structures, making them less reliable for detection.
  • Weighted Soft Voting Superiority: Among the semantic combination methods, weighted soft voting yielded the best results, highlighting the importance of segment length in the overall prediction.

Key Results

  • Overall Accuracy: The proposed model achieved an accuracy of 0.847 for the multilingual track and 0.859 for the monolingual track, outperforming the official baseline by 3.9% in the multilingual subtask.
  • Voting Method Performance: Weighted soft voting performed best, followed by soft voting, and then hard voting.
  • False Positive/Negative Rates: The hard vote approach had a higher false positive rate. The soft vote approach had a lower false positive rate but a higher false negative rate than weighted soft voting.
  • Syntactic Model Performance: The syntactic model achieved an accuracy of 49.75% and an F1 score of 33% (both micro and macro averages), indicating it was not effective for distinguishing between AI and human text.
  • Weakness in Formal Texts: The model struggled with predicting formal texts, such as those related to history, law, or academic topics.

Limitations and Future Work

  • Syntactic Approach Ineffectiveness: The authors acknowledge that their syntactic analysis using POS tags was not effective. They suggest that future work could explore Graph Neural Networks (GNNs) to analyze grammatical connections (e.g., constituency or dependency parsing trees) for a more nuanced syntactic understanding.
  • Formal Text Detection: The model's weakness in detecting formal texts indicates a need for further research and potentially specialized models or features for such content.
  • Limited Training Epochs: The semantic model was trained for only three epochs due to resource constraints, suggesting that further training could potentially improve performance.

Summary updated

BibTeX
@inproceedings{rad2024rfbes,
  author = {Mohammad Heydari Rad and Farhan Farsi and Shayan Bali and Romina Etezadi and Mehrnoush Shamsfard},
  title = {{RFBES at SemEval-2024 Task 8: Investigating Syntactic and Semantic Features for Distinguishing AI-Generated and Human-Written Texts}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure