OctavianB at SemEval-2024 Task 6: An exploration of humanlike qualities of hallucinated LLM texts

Octavian BrodoceanuNAACL 2024

TL;DR

This paper explores methods for detecting "hallucinations" – factually incorrect but fluent text generated by Large Language Models (LLMs). The authors investigate two approaches: using pre-trained models for detecting machine-generated text and leveraging the "loss" or low probability of generated sequences as a signal for hallucinations.

What Problem Does This Paper Address?

Large Language Models (LLMs) are increasingly capable of generating human-like text, but they also have a tendency to "hallucinate," producing outputs that are syntactically sound and fluent but factually incorrect or nonsensical. This poses a significant risk, especially in sensitive domains like medicine or finance, where misinformation can have severe consequences. The SHROOM shared task aims to develop methods for automatically detecting these hallucinations. This paper contributes to this effort by exploring how the characteristics of machine-generated text and the internal confidence scores of LLMs can be used to identify these problematic outputs.

Methods

The authors employed two primary methods:

  1. Generated Text Detection: This approach utilizes a pre-trained model, specifically roberta-large-openai-detector, which was trained to distinguish between human-written text and text generated by GPT-2. The hypothesis is that hallucinations, being deviations from human-like patterns, might be detectable by models trained to identify machine-generated text.
  2. GPTScore (Loss-based detection): This method leverages the concept of "GPTScore," introduced by Fu et al. (2020), which uses the negative log-likelihood (or "loss") of a generated sequence as scored by an LLM. The intuition is that sequences with low probabilities are more likely to be hallucinations. For this task, they used quantized versions of Mistral-7B and OpenHermes-13B to score the generated text. They also explored using logistic regression and SVM for binary classification based on these scores.

Datasets

The experiments were conducted on datasets from the SemEval-2024 Task 6: SHROOM. The task involves three types of text generation: Definition Modeling (DM), Paraphrase Generation (PG), and Machine Translation (MT). The datasets are divided into model-agnostic and model-aware tracks. The model-aware validation dataset contains 499 datapoints, and the model-agnostic version has 501 datapoints. Both test sets consist of 1500 samples. Human-annotated labels are used for validation and test sets, with the final label determined by a majority vote among annotators.

Key Insights and Ideas

  • Machine-generated text detectors might not be a perfect fit: While the hypothesis that machine-generated text detectors could identify hallucinations is intuitive, the results suggest a significant divergence between the tasks of general machine-generated text detection and hallucination detection.
  • LLM internal confidence is a promising signal: The GPTScore-based method, which uses the LLM's own assessment of sequence probability, showed more promising results, particularly on specific tasks. This suggests that an LLM's internal "uncertainty" can be a valuable indicator of potential factual errors.
  • Task-specific performance: The effectiveness of the methods varied significantly across different generation tasks (DM, PG, MT). Paraphrase Generation, being a more free-form task, seemed to benefit more from the tested approaches.
  • Model size matters: Larger LLMs (like OpenHermes-13B compared to Mistral-7B) generally led to improved accuracy in the GPTScore-based method, highlighting the importance of model capability in assessing text quality.

Key Results

  • Method 1 (Generated Text Detection): Achieved a score of 0.483 on the model-aware test set, which was below the baseline achieved by simply using an LLM to label generations. On the validation set, accuracy was 0.545 for the model-agnostic track and 0.465 for the model-aware track.
  • Method 2 (GPTScore):
    • On the model-aware validation set, accuracy ranged from 0.638 to 0.808 across different tasks and models.
  • On the model-agnostic validation set, accuracy ranged from 0.657 to 0.812.
  • When including the target sequence in the prompt for evaluation, accuracy reached 0.702 on the model-aware track.
  • Spearman correlation analysis indicated a near-zero correlation between the predicted probabilities and the reference labels, suggesting the method is not ideal for inferring the probability of hallucination, but rather for binary classification.
  • Task-aware results: Paraphrase Generation (PG) consistently showed higher accuracy than Definition Modeling (DM) and Machine Translation (MT) across both model-agnostic and model-aware tracks. Machine Translation performed poorly, especially on the model-agnostic track, nearing random chance.

Limitations and Future Work

  • Model limitations: The primary limitation acknowledged is the use of a pre-trained model (roberta-large-openai-detector) trained on GPT-2 generations. This model might not be sufficiently powerful to detect hallucinations from newer, larger LLMs.
  • Finetuning challenges: An attempt to finetune the roberta-large-openai-detector on a GPT-3/GPT-4 dataset resulted in a drop in performance, suggesting that current models may struggle to learn from such data or that the generations are becoming too human-like to distinguish.
  • Future Work: The authors suggest exploring model finetuning on newer datasets for human vs. machine text discrimination, using more advanced pre-trained models for text generation detection, testing the first method on other free-form generation tasks, and investigating newer methods for detecting machine-generated text that account for advancements in LLMs. Further investigation into the discrepancy in accuracy for the provided datasets when using GPTScore is also recommended.

Summary updated

BibTeX
@inproceedings{brodoceanu2024octavianb,
  author = {Octavian Brodoceanu},
  title = {{OctavianB at SemEval-2024 Task 6: An exploration of humanlike qualities of hallucinated LLM texts}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure