Kathlalu at SemEval-2024 Task 8: A Comparative Analysis of Binary Classification Methods for Distinguishing Between Human and Machine-generated Text

Lujia Cao, Ece Lara Kilic, Katharina WillNAACL 2024

TL;DR

This paper explores simple, linguistically-motivated methods for distinguishing between human and machine-generated text, finding that Zipf's Law offers surprisingly competitive performance against more complex approaches. The research highlights the potential of these intuitive methods in the face of rapidly advancing AI text generation.

What Problem Does This Paper Address?

The proliferation of powerful AI text generation models like ChatGPT has created a significant challenge: how to reliably detect machine-generated content. This is particularly critical in academic and professional settings where authenticity is paramount, and the potential for misuse (e.g., plagiarism) is high. Existing methods often rely on complex, computationally intensive models. This paper investigates whether simpler, more intuitive linguistic features can offer a viable alternative for this binary classification task.

Methods

The authors focused on two primary methods:

  • Zipf's Law: This approach leverages the observation that word frequencies in natural language follow a predictable pattern (frequency is inversely proportional to rank). The paper implements a method to fit Zipf's Law to text data by analyzing word frequencies and their ranks. The key parameter, the Zipf exponent 's', is used to characterize the text's adherence to the law. A threshold is then established based on the Zipfian distribution of human- vs. machine-generated texts to classify new samples.
  • Unigram Word Counts: This simpler method focuses on the raw count of words within a text. A threshold is determined based on the word counts of human and machine-generated texts, and new texts are classified accordingly.

Datasets

The study utilized data from SemEval-2024 Task 8, Subtask A, which focuses on monolingual English text classification. The training data included texts from various models such as ChatGPT, Cohere, Davinci, and Dolly, alongside human-generated texts. Development data featured texts from the Bloomz model and human authors. The final test data consisted of texts and their IDs.

Key Insights and Ideas

  • Simplicity Can Be Powerful: The most striking finding is that a straightforward approach based on Zipf's Law can achieve performance comparable to more sophisticated methods, including those leveraging large language models. This challenges the assumption that only complex models can tackle this problem effectively.
  • Linguistic Intuition Matters: The success of Zipf's Law suggests that fundamental linguistic properties still hold significant discriminative power, even against highly advanced generative models.
  • Zipf's Law as a Textual Fingerprint: The paper demonstrates that the adherence of a text to Zipf's Law can serve as a useful characteristic for distinguishing its origin.
  • Unigram's Limited Efficacy: While explored, the unigram word count method showed significantly lower performance compared to the Zipf's Law approach, indicating that simple frequency counts are less discriminative than distributional patterns.

Key Results

The Zipf's Law approach achieved an F1 score of 0.729 on the official leaderboard, ranking 83rd out of 137 participating teams. This performance was competitive, with the task organizers' baseline set at 0.884. The unigram approach yielded a preliminary F1 score of 0.59 on the development set, which was deemed insufficient for further development and submission.

Limitations and Future Work

The authors acknowledge that their Zipf's Law approach, while effective, did not achieve top-tier performance. They also note that time constraints prevented them from exploring the multilingual track of the task. Future work includes enhancing both the Zipf's Law and unigram systems through further literature review and addressing identified bugs in their experimental setup (specifically, an issue with a tf-idf vectorizer that caused all four tested models to produce identical predictions). They also mention exploring linear Support Vector Classifiers with character n-grams and tf-idf features, though this approach encountered unexpected issues.

Summary updated

BibTeX
@inproceedings{cao2024kathlalu,
  author = {Lujia Cao and Ece Lara Kilic and Katharina Will},
  title = {{Kathlalu at SemEval-2024 Task 8: A Comparative Analysis of Binary Classification Methods for Distinguishing Between Human and Machine-generated Text}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure