What is Artificial Intelligence? Psychometric AI as an Answer

Selmer Bringsjord, Bettina SchimanskiIJCAI 2003

Introduction

In a seminal paper presented at the 2003 International Joint Conference on Artificial Intelligence (IJCAI), Selmer Bringsjord and Bettina Schimanski proposed a provocative solution to the field's foundational "definition crisis." Rather than pursuing the ill-defined goal of "thinking" or the "gigantically ambitious" Turing Test, they argued for Psychometric AI (PAI). Under this framework, AI is defined as the field devoted to building systems that excel at established, validated psychological tests, including IQ, creativity, and aptitude benchmarks.

The Problem of Definition

The Turing Test (TT) has historically served as the gold standard for AI, yet it remains a problematic engineering goal. It is often criticized for being too vague and philosophically contentious. PAI addresses this by looking to the field of psychometrics—a discipline with over a century of experience in systematically measuring psychological properties. By adopting psychometric sub-tests as engineering specifications, AI researchers can replace abstract debates with concrete, measurable, and modular targets.

Methodology: The PAI Framework

The PAI approach utilizes a "divide and conquer" strategy based on existing psychometric categories. The authors categorize these tests into various domains, such as non-verbal abstract reasoning and physical assembly.

A standard non-verbal abstract reasoning task, similar to Raven’s Progressive Matrices, used as a benchmark in Psychometric AI.

One of the primary benchmarks used is Raven’s Progressive Matrices (RPM), which requires an agent to identify patterns in a grid of geometric shapes. Other benchmarks include the Wechsler Adult Intelligence Scale (WAIS), specifically the Block Design (physical assembly) and Comprehension (general knowledge) sub-tests.

PERI: A Case Study in PAI

To demonstrate the PAI approach, the authors developed PERI (Psychometric Experimental Robotic Intelligence). PERI is a sophisticated system integrating a Lisp-based "brain," a five-degree-of-freedom mechanical arm, and a vision system.

Central to PERI’s intelligence is a logicist approach. It utilizes theorem-prover-based agents (specifically using OTTRR) to solve problems. For 3D physical puzzles, PERI follows a general "cracking" algorithm:

  1. Document: Catalog original pieces by color and dimension.
  2. Input Goal: Decipher the target configuration from a visual input.
  3. Partition: Break the goal into pieces matching available physical blocks.
  4. Match: Use search and backtracking to determine if available pieces can satisfy the partition.
  5. Assemble: Physically manipulate the blocks into the solution.
The logicist approach in action, showing theorem-prover clauses used to deduce the solution for a complex puzzle.

Key Insights and Results

The PAI framework yields several significant insights:

  • Tests as Engineering Specifications: Standardized tests provide "clean and crisp" goals with determinate outcomes, silencing unproductive philosophical debates.
  • Speed and Accuracy: PERI demonstrates remarkable efficiency. While a human might take 30 minutes to solve a complex Binary Arts block puzzle, PERI can visually examine the blocks and deduce the solution in under a second of CPU time.
  • Infallible Reasoning: The theorem-prover agents are capable of "infallibly cracking" geometric analogies and RPM items they have never encountered before.
  • Weak AI Alignment: The authors align PAI with "Weak AI," arguing that if a machine appears intelligent by passing validated tests, it should be treated as such, regardless of internal consciousness.

Limitations and Future Work

While PERI excels at logic-based and physical tasks, the authors acknowledge challenges in more narrative-driven tests. The "Picture Arrangement" sub-test, which requires ordering snapshots to tell a coherent story, remains a significant hurdle. Future work involves using narrative constraints and consistency proofs to tackle these more complex, "common sense" benchmarks. Furthermore, the authors suggest that the logic developed for these tests could have real-world applications in fields like threat anticipation, where analysts must reconstruct "stories" from fragmented data points.

Conclusion

Psychometric AI offers a rigorous, operationalized path forward for the field. By grounding AI development in the established science of mental measurement, researchers can move beyond the limitations of the Turing Test toward building systems with broad, measurable, and verifiable intelligence.

Summary updated

BibTeX
@inproceedings{bringsjord2003what,
  author = {Selmer Bringsjord and Bettina Schimanski},
  title = {{What is Artificial Intelligence? Psychometric AI as an Answer}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2003},
}
Figure