TM-TREK at SemEval-2024 Task 8: Towards LLM-Based Automatic Boundary Detection for Human-Machine Mixed Text

Xiaoyan Qu, Xiangfeng MengNAACL 2024

TL;DR

This paper tackles the growing challenge of distinguishing between human-written and LLM-generated text, specifically focusing on "mixed texts" that blend both. They frame this as a token-level boundary detection problem and achieve state-of-the-art results by fine-tuning LLMs and using an ensemble approach, even winning a SemEval competition.

What Problem Does This Paper Address?

The rapid advancement of Large Language Models (LLMs) like ChatGPT has made it increasingly difficult to discern between human-authored and machine-generated content. While previous work focused on classifying entire texts as either human or LLM-generated, this paper addresses a more nuanced and increasingly common scenario: "mixed texts." These texts seamlessly integrate human writing with LLM-generated portions, posing a significant challenge for traditional classification methods. The ability to accurately identify the transition points (boundaries) in such texts is crucial for combating issues like misinformation, academic dishonesty, and ensuring the integrity of digital content.

Methods

The core of this work is framing the problem of boundary detection in human-machine mixed texts as a token classification task. Each token in the text is assigned a label indicating whether it's human-written or LLM-generated. The transition point between these labels signifies the boundary.

Key methodological aspects include:

  • LLM Fine-tuning: The researchers leverage LLMs known for their ability to handle long-range dependencies, specifically Longformer, XLNet, and BigBird. These models are fine-tuned on the boundary detection task.
  • Ensemble Strategy: To boost performance, they employ an ensemble of multiple fine-tuned LLMs, combining their predictions (specifically, a voting mechanism on output logits) to achieve a more robust result. This approach was instrumental in their SemEval win.
  • Exploration of Factors: The paper investigates several factors that influence LLM performance in this task:
  • Extra Layers: Adding supplementary layers like LSTM and BiLSTM on top of LLMs.
  • Segment Loss Functions: Adapting loss functions from image segmentation (like Dice loss) to better capture the significance of tokens near the boundary. They also propose a novel BCE-MAE loss.
  • Pretraining: Utilizing data from related tasks (binary and multi-way text classification) for pretraining LLMs before fine-tuning them on the boundary detection task. Two distinct pretraining strategies are explored.

Datasets

The primary dataset used is an extension of the M4 dataset, consisting of 3,649 training cases and 505 development cases, each with text content and a gold boundary index.

Additionally, the paper mentions using two supplementary datasets from SemEval'24 Task 8 related to binary and multi-way text classification for pretraining purposes. These datasets contain a significantly larger number of examples (over 119,000 for binary and over 71,000 for multi-way classification).

Key Insights and Ideas

  • Mixed Text Detection is Crucial: The paper highlights that simply classifying text as "human" or "LLM" is insufficient; detecting boundaries in mixed texts is a critical next step.
  • LLMs Excel at Boundary Detection: LLMs, particularly those adept at long-range dependencies, show significant promise in identifying these transition points.
  • Ensembling is Powerful: Combining predictions from multiple LLMs (specifically XLNet models) significantly improves boundary detection accuracy, as evidenced by their competition win.
  • Loss Functions Matter: Adapting loss functions from image segmentation (like Dice loss) can effectively improve performance by focusing on the importance of tokens near the boundary.
  • Pretraining Boosts Performance: Pretraining on related text classification tasks can enhance LLMs' ability to detect token-wise boundaries in mixed texts.

Key Results

  • The authors' ensemble approach achieved first place in the SemEval'24 Competition Task 8, Sub-task C ('Human-Machine Mixed Text Detection').
  • XLNet-large achieved the best individual LLM performance with a Mean Absolute Error (MAE) of 2.44.
  • An ensemble of two XLNet models with a voting strategy further reduced the MAE to 2.22.
  • Adding BiLSTM layers to Longformer-large resulted in a significant MAE decrease from 3.58 to 2.73.
  • The proposed BCE-MAE loss function showed a notable improvement, reducing MAE by 16.48% compared to the baseline.
  • Pretraining with sentence-level data (Pretrain 1) combined with BiLSTM layers achieved an MAE of 2.84.

Limitations and Future Work

  • Limitations: The paper notes that adding a CRF layer to Longformer-large did not yield satisfactory results, possibly due to the lack of clear dependencies between the binary labels (0 and 1) in this specific task, unlike in named entity recognition.
  • Future Work: The authors suggest that future research can further advance LLMs' capabilities in detecting boundaries within mixed texts. The paper establishes a state-of-the-art benchmark for this new dataset, paving the way for further exploration.

Summary updated

BibTeX
@inproceedings{qu2024tm,
  author = {Xiaoyan Qu and Xiangfeng Meng},
  title = {{TM-TREK at SemEval-2024 Task 8: Towards LLM-Based Automatic Boundary Detection for Human-Machine Mixed Text}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure