Does GPT-4 pass the Turing test?

Cameron R. Jones, Ben BergenNAACL 2024 · DOI

TL;DR

This paper reports on a large-scale online Turing test where participants interacted with GPT-4 and other AI models, as well as humans. The results show that while GPT-4, with a carefully crafted prompt, can fool humans nearly 50% of the time, it still falls short of human performance and doesn't consistently pass the Turing test. This highlights that linguistic style and socio-emotional traits are key factors in human-like communication, rather than just raw intelligence.

What Problem Does This Paper Address?

The Turing test, proposed by Alan Turing in 1950, is a classic benchmark for artificial intelligence, aiming to determine if a machine can exhibit intelligent behavior equivalent to, or indistinguishable from, that of a human. Despite its conceptual significance, the test has faced numerous criticisms regarding its validity as a measure of true intelligence. This paper addresses the question of whether modern large language models (LLMs), specifically GPT-4, can pass the Turing test in a real-world, public setting. It also explores the factors that contribute to successful deception and the strategies interrogators use to detect AI.

Methods

The researchers conducted a large-scale online Turing test using a platform called turingtest.live. They created 45 "LLM Witnesses" using OpenAI's API, varying GPT-3.5 and GPT-4 models, temperature settings, and carefully designed prompts. These prompts instructed the AI on how to behave, including whether to make spelling errors, adopt a specific personality, and how to respond to various situations. The AI witnesses were also programmed with delays to mimic human typing.

Participants (interrogators) were randomly assigned to interact with either a human or an AI witness for 5 minutes. After the conversation, they had to decide if their partner was human or AI, providing a confidence score and a reason for their decision. The study collected data from 1979 participants over 6845 games.

Datasets

The paper does not explicitly mention training datasets for the LLMs, as they used pre-trained models (GPT-3.5 and GPT-4) from OpenAI. The evaluation was based on the interactions within the turingtest.live platform, which generated its own dataset of conversations and participant decisions.

Key Insights and Ideas

  • GPT-4 is getting close, but not quite there: The best-performing GPT-4 prompt achieved a success rate of 49.7%, meaning it was misidentified as human nearly half the time. This is a significant improvement over previous models but still below the human baseline.
  • Linguistic style and socio-emotional traits matter more than raw intelligence: Participants' decisions were primarily based on linguistic style (35%) and socio-emotional traits (27%), rather than deep knowledge or reasoning abilities. This suggests that mimicking human conversational nuances is crucial for passing the Turing test.
  • Prompt engineering is key: The performance of GPT-4 varied significantly based on the prompt used, highlighting the importance of carefully crafting instructions for LLMs to achieve desired behaviors.
  • Experience and knowledge improve detection: Interrogators who had more experience playing the game and possessed greater knowledge about LLMs were more accurate in distinguishing between humans and AI. This suggests that practice and understanding of AI capabilities can help mitigate deception.
  • ELIZA still holds its own (in a way): Surprisingly, the older ELIZA chatbot performed better than some GPT-3.5 and even some GPT-4 witnesses, suggesting that its simplicity and rule-based approach could be less "detectable" as AI than more sophisticated but sometimes "unnatural" LLMs.

Key Results

  • Human witnesses achieved a 66% success rate.
  • The best-performing GPT-4 prompt ("Dragon") achieved a 49.7% success rate across 855 games, which was not statistically significantly different from a 50% chance baseline.
  • GPT-3.5 models performed worse, with success rates ranging from 5% to 20%.
  • ELIZA achieved a 22% success rate, outperforming several GPT-3.5 and GPT-4 witnesses.
  • Reasons for AI verdicts were predominantly linguistic (e.g., too formal/informal, grammar too perfect) and socio-emotional (e.g., lack of personality, unnatural responses).
  • Reasons for human verdicts also included linguistic and socio-emotional factors, but often involved positive attributes like humor and a plausible backstory.

Limitations and Future Work

The authors acknowledge several limitations:

  • Biased Sample: Participants were recruited via social media, potentially leading to a non-representative sample.
  • Lack of Incentives: Participants were not incentivized, which might have affected their motivation and performance.
  • Human Witnesses "Trolling": Some human witnesses pretended to be AI, potentially skewing results.
  • Interrogator Familiarity: Some interrogators knew their witnesses personally.
  • AI Witness Bias: Participants sometimes had an a priori belief that they were playing against an AI due to repeated matches.
  • Closed-Source Models: The use of OpenAI's models limits reproducibility.
  • Limited Prompt Exploration: The study tested a relatively small number of prompts, suggesting that more effective ones likely exist.

Future work suggested includes:

  • Conducting studies in more controlled settings with pre-registration of systems and criteria.
  • Exploring other LLMs and giving models access to external tools like web browsing.
  • Investigating the impact of different prompt engineering techniques more systematically.
  • Testing models fine-tuned with reinforcement learning.

Summary updated

BibTeX
@inproceedings{jones2024does,
  author = {Cameron R. Jones and Ben Bergen},
  title = {{Does GPT-4 pass the Turing test?}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.290},
}
Figure