HU at SemEval-2024 Task 8A: Can Contrastive Learning Learn Embeddings to Detect Machine-Generated Text?

Shubhashis Roy Dipta, Sadat ShahriarNAACL 2024

TL;DR

This paper explores a novel approach to detecting machine-generated text using contrastive learning, achieving comparable performance to larger, ensemble-based models with significantly fewer parameters. Their method, which includes a unique data augmentation technique, demonstrates the potential of a single, model-agnostic architecture for this challenging task.

What Problem Does This Paper Address?

The proliferation of large language models (LLMs) has made it increasingly difficult to distinguish between human-written and machine-generated text. This poses significant challenges in areas like academic integrity, cybersecurity (phishing), and content authenticity. Existing detection methods often rely on knowing the specific LLM used for generation, which is impractical in real-world "black-box" scenarios. The SemEval-2024 Task 8A specifically addresses this by providing a dataset with texts generated by multiple LLMs across different domains and languages, demanding a versatile and model-agnostic detection system.

Methods

The core of the proposed system is a contrastive learning framework. The authors' main hypothesis is that machine-generated text and human-generated text will have distinct embedding representations. They employ a single unified model (specifically, a Longformer-base encoder) to generate these embeddings.

A key innovation is their data augmentation technique. To create the positive and negative pairs required for contrastive learning, they paraphrase existing texts. Instead of paraphrasing entire documents, which can lead to information loss and text shortening, they split documents into sentences, paraphrase each sentence individually, and then reassemble them. This effectively expands the training data.

For the contrastive loss, they use a cosine similarity-based formulation. The overall model loss is a combination of this contrastive loss and a binary cross-entropy (BCE) loss from a simple two-layer classifier head. This classifier head is applied to the embeddings generated by the encoder.

Datasets

The primary dataset used is from the SemEval-2024 Task 8A. This dataset includes machine-generated texts from five different LLMs and human-generated texts across various domains and languages. The authors used the official train and development splits for training and evaluation, respectively. Data augmentation was performed only on the training set.

Key Insights and Ideas

  • Contrastive Learning as a Model-Agnostic Solution: The paper strongly suggests that contrastive learning can learn embeddings that effectively differentiate machine-generated text from human text, even without prior knowledge of the specific generation model. This is crucial for real-world applications.
  • Data Augmentation is Key: The proposed sentence-level paraphrasing technique significantly boosts the training data, enabling the model to generalize better. This is highlighted as a critical component for achieving comparable performance with a single model.
  • Simplicity Can Be Effective: The authors demonstrate that a single, relatively smaller model (≈40% of the baseline's parameters) can achieve competitive results, challenging the notion that complex ensemble methods are always necessary for high performance.
  • Dropout's Role: Surprisingly, a high dropout rate (60%) in the classifier head did not improve performance and, in some cases, degraded it. This suggests that contrastive learning and data augmentation are the primary drivers of generalization, rather than extensive regularization in the classifier.
  • Batch Size Matters: The study found that a very small effective batch size (2) yielded superior results compared to larger batch sizes achieved through gradient accumulation. This indicates that the interaction between samples within a batch might be more critical than the sheer volume of data processed per step in this specific contrastive learning setup.

Key Results

The paper reports that their final submission, Ourscon, achieved a comparable performance to the official baseline model (baselinerob) on the test dataset, ranking 21st out of 137 participants. Notably, their model used significantly fewer parameters (149M vs. 355M).

Through hyperparameter tuning (Ourscon+), they achieved an accuracy of 93.36% with a maximum sentence length of 256, a significant improvement over the baseline and their initial submission. This performance gain, achieved with a single model, underscores the effectiveness of their approach.

Limitations and Future Work

Limitations: * The paper acknowledges that the paraphrasing process, while effective, can sometimes shorten texts, which was an observed issue with their chosen paraphrase model. * The study used a fixed effective batch size of 2, which, while yielding the best results, might be computationally inefficient for larger-scale training. * The authors mention that using multiple models or prompt-based models for data augmentation was left out due to time and compute constraints, suggesting potential for further improvement.

Future Work: * Explore recent prompt-based models for more advanced data augmentation. * Investigate alternative contrastive loss functions, such as Triplet loss or InfoNCE loss, to potentially enhance embedding quality. * Further explore the impact of different batch sizes and gradient accumulation strategies.

Summary updated

BibTeX
@inproceedings{dipta2024hu,
  author = {Shubhashis Roy Dipta and Sadat Shahriar},
  title = {{HU at SemEval-2024 Task 8A: Can Contrastive Learning Learn Embeddings to Detect Machine-Generated Text?}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure