AI Generality and Spearman's Law of Diminishing Returns

José Hernández-Orallo

TL;DR

This paper investigates whether AI systems exhibit a "positive manifold" – a correlation between performance across different tasks – and whether this generality follows Spearman's Law of Diminishing Returns (SLODR), which suggests that general intelligence becomes less important for higher-ability individuals. The authors find that SLODR does not appear in AI systems, suggesting that generality might behave differently in artificial agents compared to humans.

What Problem Does This Paper Address?

The field of AI is increasingly focused on developing general-purpose AI systems that can perform well across a wide range of tasks, rather than specializing in a single one. This trend is reflected in the design of benchmarks and competitions that feature numerous diverse tasks. However, it's unclear if the best-performing AI systems are indeed the most general, or if the pursuit of generality is always beneficial. This question mirrors a long-standing debate in human intelligence research concerning the "positive manifold" and Spearman's Law of Diminishing Returns (SLODR). The positive manifold suggests that performance on different cognitive tasks is correlated, implying a general intelligence factor (g). SLODR, a controversial aspect of this, posits that the influence of this general factor diminishes for individuals with higher abilities. This paper aims to explore these concepts in the context of AI, investigating whether AI systems exhibit a similar positive manifold and if SLODR applies to them.

Methods

The authors employ two main approaches to investigate AI generality:

  1. Empirical Study on a General Video Game AI (GVGAI) Competition: They analyze data from the 2015 GVGAI competition, where AI agents (controllers) compete across a variety of games (tasks). This involves calculating correlations between agent performance on different games to assess the presence of a positive manifold. They then analyze these correlations across different ability bins of agents to check for SLODR. They also explore the impact of task difficulty on these correlations.
  2. Controlled Synthetic Setting with Elementary Cellular Automata (ECA): To gain more control over the experimental conditions, they create a synthetic environment using ECA. In this setting, randomly generated agents interact with ECA rules (tasks). This allows for a more systematic exploration of agent populations and task spaces, enabling a detailed analysis of generality and SLODR under controlled conditions. They experiment with different reward mechanisms and task difficulty metrics.

Datasets

  • GVGAI Competition Data (2015): This dataset consists of results from 23 AI agents competing across 49 games, each with 5 levels, totaling 245 tasks. Performance is measured by win/loss ratios.
  • Synthetic ECA Environment: This involves 256 elementary cellular automaton rules (tasks) and 400 randomly generated agent programs (agents) in one experiment, and 256 agents in another. Performance is measured by reward signals.

Key Insights and Ideas

  • AI vs. Human Generality: The most striking insight is that Spearman's Law of Diminishing Returns (SLODR) does not appear to hold for AI systems in the studied scenarios. This suggests that generality might behave fundamentally differently in artificial agents compared to humans.
  • Weak Positive Manifold in AI: While a positive manifold (correlation across tasks) is observed in the GVGAI competition, it is generally weak. In the synthetic ECA setting, the positive manifold is even weaker, sometimes almost non-existent.
  • The Role of Task Difficulty: The paper highlights the importance of task difficulty in analyzing generality. Simply slicing tasks by a predefined "level" in GVGAI proved less effective than an empirically derived difficulty metric.
  • No Universal Law of Augmenting Returns (ULOAR) either: While SLODR doesn't appear, the data doesn't strongly support its opposite, a Universal Law of Augmenting Returns (ULOAR), where generality consistently increases with ability. The trends are often timid or inconclusive.
  • Generality as a "Second-Best" Trait: The paper frames generality as being "second-best" at everything, contrasting with "idiots savants" who excel at very specific tasks. This perspective is applied to AI agents.

Key Results

  • GVGAI Competition:
    • An average correlation of 0.106 was found across all tasks and agents, indicating a weak positive manifold.
  • Analysis of ability bins did not reveal a diminishing trend for the positive manifold; SLODR was not observed.
  • Slicing tasks by empirically derived difficulty showed some slight trends, but overall, SLODR was not supported.
  • Synthetic ECA Setting:
    • With randomly generated agents and tasks, the average correlation was consistently low (e.g., 0.125 for the initial setup, dropping to 0.004 with mirrored rewards).
  • In most ECA experiments, SLODR was not observed. In some cases, there was a slight indication of increasing correlation with ability, hinting at a potential ULOAR, but this was not statistically robust.
  • Factor analysis confirmed the weak presence of a general factor in these scenarios.

Limitations and Future Work

  • Small Sample Sizes: The authors acknowledge that the GVGAI study used a relatively small population of agents (23) and tasks (245). The synthetic ECA experiments also involved limited agent populations in some analyses.
  • Difficulty Metric: The initial "level" metric for task difficulty in GVGAI was found to be inadequate, requiring an empirical derivation. The synthetic setting's difficulty metric also had simplifications.
  • Task Order Irrelevance: The experiments assume tasks are presented independently, which might not reflect real-world scenarios where task order or transfer learning could influence generality.
  • Future Work:
    • Principled Difficulty Estimation: The authors advocate for more principled approaches to estimating task difficulty, potentially based on policy descriptions rather than average performance.
  • Ratio Scales for Performance: Developing methods to obtain ratio scales for task performance would allow for more meaningful aggregation of results.
  • More Abstract Tasks: Exploring whether generality can be measured with fewer, more abstract tasks, requiring a deeper theoretical understanding of what tasks measure.
  • Hyper-agents and Meta-Learning: Investigating if "hyper-agents" (systems that select specialized sub-procedures) increase the general factor in AI systems.
  • Multi-Agent and Competitive Scenarios: Further analysis of generality in competitive settings, where "social intelligence" and consistent performance against diverse opponents are key.
  • Longitudinal Studies: Analyzing how generality indicators evolve in competitions over time.

Summary updated

Abstract

Many areas of AI today use benchmarks and competitions with larger and wider sets of tasks. This tries to deter AI systems (and research effort) from specialising to a single task, and encourage them to be prepared to solve previously unseen tasks. It is unclear, however, whether the methods with best performance are actually those that are most general and, in perspective, whether the trend moves towards more general AI systems. This question has a striking similarity with the analysis of the so-called positive manifold and general factors in the area of human intelligence. In this paper, we first show how the existence of a manifold (positive average pairwise task correlation) can also be analysed in AI, and how this relates to the notion of agent generality, from the individual and the populational points of view. From the populational perspective, we analyse the following question: is this manifold correlation higher for the most or for the least able group of agents? We contrast this analysis with one of the most controversial issues in human intelligence research, the so-called Spearman's Law of Diminishing Returns (SLODR), which basically states that the relevance of a general factor diminishes for most able human groups. We perform two empirical studies on these issues in AI. We analyse the results of the 2015 general video game AI (GVGAI) competition, with games as tasks and "controllers" as agents, and the results of a synthetic setting, with modified elementary cellular automata (ECA) rules as tasks and simple interactive programs as agents. In both cases, we see that SLODR doesnot appear. The data, and the use of just two scenarios, does not clearly support the reverse either, a Universal Law of Augmenting Returns (ULOAR), but calls for more experiments on this question.
BibTeX
@article{hernandezorallo2019ai,
  author = {José Hernández-Orallo},
  title = {{AI Generality and Spearman's Law of Diminishing Returns}},
  journal = {J. Artif. Intell. Res.},
  year = {2019},
  doi = {10.1613/jair.1.11388},
}
Figure