Twenty Years Beyond the Turing Test: Moving Beyond the Human Judges Too

José Hernández-Orallo

TL;DR

This paper argues that the Turing Test, while historically significant, is no longer a sufficient or even relevant benchmark for evaluating artificial intelligence. It proposes a shift towards machine-based, adversarial evaluation frameworks that move beyond human-centric metrics and references, enabling a more robust and scalable assessment of AI capabilities.

What Problem Does This Paper Address?

The paper tackles the obsolescence of the Turing Test as a primary evaluation metric for AI. While the Turing Test aimed to assess if a machine could exhibit human-like conversational behavior, the authors argue that AI has advanced to a point where this test is insufficient. The core problem is the reliance on human judges and human performance as the ultimate benchmark, which limits our understanding of AI's potential and creates an "abyss beyond superhuman performance." This anthropocentric view hinders the development of truly novel AI capabilities and creates a "challenge-solve-and-replace" dynamic in AI benchmarks, where new benchmarks are constantly needed as AI surpasses human performance. The paper also addresses the "cognitive-judge problem," where human judges struggle to consistently and reliably evaluate complex AI behaviors, especially in generative tasks.

Methods

The paper proposes two new evaluation schemas derived from the principles of "Turing learning" (a paradigm involving generators and discriminators, exemplified by GANs):

  • Turing Testing: This schema replaces human judges with machine discriminators. The discriminator adapts its queries to both a real entity and an AI imitator, learning to distinguish between them. This process is adversarial, as the discriminator aims to improve its judgment while the imitator (or testee) aims to fool it.
  • Adversarial Testing: This schema further refines Turing Testing by removing the "imitator" role and focusing solely on an adversarial interaction between a "testee" (the AI being evaluated) and a "tester" (the machine judge). The tester adapts its questions to find the most informative instances for assessing the testee's abilities, moving towards an ability-oriented evaluation rather than a task-oriented one. This approach is inspired by computerised adaptive testing.

Datasets

The paper does not introduce new datasets. It references existing benchmarks and datasets in the context of AI evaluation, such as:

  • CIFAR-10: Used as an example of an image recognition benchmark where AI performance is tracked against human levels.
  • Pac-Man: Used as an example of a game where AI performance can exceed human scores, highlighting the limitations of score-based metrics.
  • ImageNet: Mentioned as an example of a dataset where transitional problems (e.g., adding noise and blur) can be created to extend evaluation.
  • Winograd Schema Challenge: Mentioned as a benchmark for commonsense reasoning.
  • CAPTCHAs: Referenced as an example of automated tests.

Key Insights and Ideas

  • The Turing Test is a Relic: The paper strongly argues that the Turing Test is outdated and no longer an effective measure of intelligence due to AI's progress and the inherent limitations of human judges.
  • Machine Judges are the Future: Replacing human judges with machine discriminators (in Turing Testing) and testers (in Adversarial Testing) is crucial for scalable, objective, and robust AI evaluation.
  • Beyond Human-Centricity: AI evaluation needs to move beyond anthropocentric benchmarks and references. AI capabilities can and should exceed human levels, and evaluation should reflect this.
  • Turing Learning as a Foundation: The principles of Turing learning, with its generator-discriminator dynamic, provide a strong foundation for developing new adversarial testing paradigms.
  • Adversarial Testing for Deeper Understanding: Adversarial testing, by focusing on finding informative instances rather than just passing/failing, can better probe deeper AI capabilities like understanding and meaning extraction.
  • Behavioral Taxonomies: The proposed testing frameworks can be used to build "behavioral taxonomies," mapping and comparing the intelligence of diverse systems (AI, animals, humans) in a multidimensional space.

Key Results

The paper is primarily theoretical and proposes new frameworks rather than presenting empirical results from specific experiments. The "results" are conceptual:

  • Conceptual Frameworks: The paper successfully outlines and defines two novel evaluation frameworks: Turing Testing and Adversarial Testing.
  • Identification of Evaluation Categories: The authors categorize AI evaluation scenarios into "Ceiling," "Projectional," "Transitional," and "Universal" to illustrate the challenges of extrapolating beyond human performance.
  • Argument for Machine Judges: The paper builds a strong case for why machine judges are superior to human judges for AI evaluation, citing issues of scalability, objectivity, and the ability to adapt and learn.

Limitations and Future Work

  • Implementation Challenges: While the frameworks are proposed, the paper acknowledges that significant work is needed to implement them effectively, particularly in designing appropriate "testers" and "testees" and defining relevant difficulty metrics.
  • Defining "Understanding" and "Meaning": The paper raises the question of whether machine judges can truly evaluate abstract concepts like "understanding" and "meaning" if they themselves lack these capabilities. This remains an open philosophical and technical challenge.
  • Computational Effort: The paper notes the need to analyze the computational effort required for testers versus testees, especially in complex domains.
  • Building Behavioral Taxonomies: The development of robust behavioral taxonomies based on these new testing methods is presented as a significant area for future research.
  • AI for Cognitive Measurement: The paper suggests that AI itself can be a powerful tool for cognitive measurement and comparative cognition, and that this is an area ripe for further exploration.
  • Human Augmentation: The long-term implication of AI-driven cognitive enhancers on human evolution and behavior is also highlighted as a future research direction.

Summary updated

Abstract

In the last 20 years the Turing test has been left further behind by new developments in artificial intelligence. At the same time, however, these developments have revived some key elements of the Turing test: imitation and adversarialness. On the one hand, many generative models, such as generative adversarial networks (GAN), build imitators under an adversarial setting that strongly resembles the Turing test (with the judge being a learnt discriminative model). The term “Turing learning” has been used for this kind of setting. On the other hand, AI benchmarks are suffering an adversarial situation too, with a ‘challenge-solve-and-replace’ evaluation dynamics whenever human performance is ‘imitated’. The particular AI community rushes to replace the old benchmark by a more challenging benchmark, one for which human performance would still be beyond AI. These two phenomena related to the Turing test are sufficiently distinctive, important and general for a detailed analysis. This is the main goal of this paper. After recognising the abyss that appears beyond superhuman performance, we build on Turing learning to identify two different evaluation schemas: Turing testing and adversarial testing. We revisit some of the key questions surrounding the Turing test, such as ‘understanding’, commonsense reasoning and extracting meaning from the world, and explore how the new testing paradigms should work to unmask the limitations of current and future AI. Finally, we discuss how behavioural similarity metrics could be used to create taxonomies for artificial and natural intelligence. Both testing schemas should complete a transition in which humans should give way to machines—not only as references to be imitated but also as judges—when pursuing and measuring machine intelligence.
BibTeX
@article{hernandezorallo2020twenty,
  author = {José Hernández-Orallo},
  title = {{Twenty Years Beyond the Turing Test: Moving Beyond the Human Judges Too}},
  journal = {Minds Mach.},
  year = {2020},
  doi = {10.1007/s11023-020-09549-0},
}
Figure