Reliability Check: An Analysis of GPT-3’s Response to Sensitive Topics and Prompt Wording

Aisha Khatun, Daniel BrownACL 2023

TL;DR

This paper investigates the reliability of GPT-3 when responding to sensitive topics and how prompt wording affects its answers. The authors found that while GPT-3 can correctly identify obvious conspiracies and stereotypes, it struggles with misconceptions and controversies, often providing inconsistent responses across different prompts. This highlights the need for careful prompt engineering and further research into LLM reliability.

What Problem Does This Paper Address?

Large Language Models (LLMs) like GPT-3 have become incredibly powerful and versatile, but their reliability, especially on sensitive or debatable topics, remains a significant concern. While efforts like Reinforcement Learning with Human Feedback (RLHF) aim to improve neutrality and safety, there's a lack of systematic analysis into how these models behave across different categories of statements and how subtle changes in prompts can lead to drastically different or even contradictory outputs. This paper aims to fill that gap by analyzing GPT-3's responses to various sensitive topics and exploring the impact of prompt wording on its reliability.

Methods

The researchers curated a dataset of 1268 statements across six categories: Fact, Conspiracy, Controversy, Misconception, Fiction, and Stereotype. They then designed four distinct prompts to query GPT-3 about these statements. These prompts varied in their directness and framing, ranging from a simple "Is this true?" to more nuanced phrasing like "In the real world?" and "Scientific Knowledge." The core of their method involved analyzing both the binary YES/NO token responses and the full text generated by GPT-3, along with the confidence scores associated with these responses. They compared these outputs against known ground truth where available and analyzed inconsistencies across different prompts for the same statement.

Datasets

The dataset consists of 1268 statements collected from various sources, including:

  • GPT-3 itself: Used to generate some initial statements.
  • Wikipedia: Lists of misconceptions, conspiracy theories, and controversies.
  • Conspiracy Theory Papers: Academic and popular articles on conspiracy theories.
  • External Links: Websites and articles related to the categories.
  • Books: A small contribution.

The paper does not introduce a new benchmark dataset but rather curates and categorizes existing information to test LLM responses.

Key Insights and Ideas

  • Prompt Sensitivity is High: Even minor changes in prompt wording can lead to significant shifts in GPT-3's responses, confidence levels, and even its factual conclusions. This suggests LLMs are highly susceptible to framing.
  • Confusion on Nuance: GPT-3 struggles most with "Misconception" and "Controversy" categories, where ground truth is either unknown to the public or has competing viewpoints. This indicates a difficulty in navigating ambiguity and widely held false beliefs.
  • RLHF's Double-Edged Sword: While RLHF has made GPT-3 better at disagreeing with negative stereotypes and aligning with powerful entities, it can also lead to an over-reliance on neutrality, sometimes causing it to dismiss factual information or provide overly cautious responses.
  • Inconsistent "Truthfulness": The model can sometimes provide a YES/NO answer that contradicts its own full-text explanation, or even contradict itself across different prompts for the same statement. This highlights a lack of deep understanding and reliance on pattern matching.
  • "I think..." Prompting is Volatile: The "I think..." prompt, designed to elicit personal opinion, often leads to agreeable but factually incorrect responses, demonstrating how easily the model can be swayed by user input.

Key Results

  • High Confidence, Low Reliability: Across most categories, GPT-3 responses exhibit high confidence scores (often >97.5%), even when incorrect.
  • Category Performance:
    • Facts: Mostly correctly identified, but some obvious facts still show inconsistencies across prompts.
  • Conspiracies: Generally correctly identified as false (ground truth is NO), with high confidence in NO responses.
  • Misconceptions & Controversies: Show the most confusion, with a higher number of statements having less than 90% confidence and more prompt response conflicts.
  • Stereotypes: GPT-3 is highly confident, often agreeing with positive stereotypes and disagreeing with negative ones, likely due to RLHF.
  • Fiction: Responses can be inconsistent, with different prompts yielding different answers for the same fictional statement.
  • Prompt Impact: Table 2 shows that Prompt 0 ("Is this true?") resulted in the most conflicts with ground truth, while Prompt 2 ("Scientific Knowledge") had the fewest.
  • Response Changes: Approximately 6.2% of statements saw a change in GPT-3's YES/NO answer solely due to prompt variations.

Limitations and Future Work

Limitations: * Manual Analysis: The study relied on manual analysis of full-text responses, which is time-consuming and could be expanded. * Limited Prompts: Only four prompts were used, and a more extensive set could reveal further nuances. * Model Version: The study used text-davinci-003, and newer models like ChatGPT API and GPT-4 were not analyzed.

Future Work: * More Nuanced Analysis: Further analysis of model outputs against categories and prompts is planned. * Dataset Refinement: Cleaning the dataset to remove unexpected confusions and adding more ground truth labels. * Broader Model Testing: Extending the study to include newer LLMs and a wider range of prompt variations. * Ethical Standards: Developing techniques to better define and enforce LLM ethical standards and ensure systematic understanding of how different techniques interact.

Summary updated

BibTeX
@inproceedings{khatun2023reliability,
  author = {Aisha Khatun and Daniel Brown},
  title = {{Reliability Check: An Analysis of GPT-3’s Response to Sensitive Topics and Prompt Wording}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure