TL;DR
This paper proposes a novel framework to disentangle "generality" and "capability" as distinct metrics for intelligence, moving beyond population-level analyses. By introducing agent characteristic curves (ACCs) and defining capability as the area under the curve and generality as its "compactness" on the difficulty axis, they offer a quantitative tool to assess intelligence in humans, animals, and AI, paving the way for more nuanced evaluations of AI progress.
What Problem Does This Paper Address?
The traditional understanding of "general intelligence" has been a somewhat nebulous concept, often interpreted as success across a wide range of situations. However, this has led to difficulties in precisely measuring and comparing intelligence, especially across different species or between natural and artificial agents. Existing approaches, like the "g factor," are population-based and don't offer individual-level insights. Furthermore, the interplay between an agent's overall performance (capability) and its breadth of performance across varying task difficulties (generality) has been entangled. This paper aims to disentangle these two crucial aspects, providing a more rigorous and individual-centric way to quantify intelligence.
Methods
The core of the paper's methodology lies in defining and measuring two new metrics: capability and generality.
- Agent Characteristic Curves (ACCs): For each agent (individual or group), their performance is plotted against task difficulty, creating an ACC. This curve visually represents how an agent's performance changes as tasks become harder.
- Capability ($\Psi_j$): This is defined as the area under the ACC. It represents the agent's overall performance across the entire range of difficulties.
- Spread ($S_j$) and Generality ($\Gamma_j$): Generality is derived from "spread," which measures how much an agent's performance deviates from an ideal step-like curve (high generality). A compact curve on the left (performing well on easy tasks and dropping off sharply on harder ones) indicates high generality. Generality is then defined as the reciprocal of spread.
- Difficulty Metric ($\hbar$): A crucial aspect is the need for a well-defined measure of task difficulty. This can be intrinsic (derived from task properties) or extrinsic (derived from population performance or reference agents).
Datasets
The paper applies its framework to various existing datasets and benchmarks, including:
- Elithorn's mazes: A perceptual maze test used to study problem-solving skills.
- Letter series: A classic intelligence test, with instances generated using Levin's Kt for difficulty.
- Object Recognition: Data from a study comparing humans, macaques, and ANNs on object classification with psychophysically derived difficulty.
- Odour Span Task (OST): Data from rats on an olfactory working memory task.
- Iris flower classification: Using classifiers from OpenML with intrinsic difficulty metrics (KDN and TDU).
- Chess competitions (Reykjavik 2005, Leiden 2015): Using opponent's average performance as difficulty.
- ALE video games: Using AI systems and human references.
- GVGAI video games: Using AI systems.
- Physical Cognition: Data from orangutans.
- Primate Cognition Test Battery (PCTB): Data from humans, chimpanzees, and orangutans.
The paper does not introduce new datasets but rather re-analyzes existing ones through the lens of its new metrics.
Key Insights and Ideas
- Generality and Capability Decouple: A major finding is that an agent can be highly capable but not very general (a specialist) or very general but not highly capable (a "generic assistant"). This challenges the traditional conflation of these two concepts.
- Individual-Level Metrics: The proposed metrics allow for the assessment of generality and capability at the individual level, moving beyond population-level statistics like the "g factor." This is particularly important for evaluating AI systems where natural populations don't exist.
- Difficulty is Key: The choice and definition of a difficulty metric are paramount. The framework highlights how different difficulty measures can lead to different interpretations of an agent's performance.
- Challenging SLODR: The paper provides evidence that the "Spearman's Law of Diminishing Returns" (SLODR), which suggests generality decreases with increasing capability, does not hold when generality and capability are properly disentangled and measured. In many cases, higher capability correlates with higher generality.
- Generality as "Compactness": High generality is intuitively linked to an agent's performance being "compact" on the easier end of the difficulty spectrum, meaning it performs well across a wide range of easy tasks and drops off more predictably for harder ones.
Key Results
- Humans Tend to Be More General: Across several experiments, humans generally exhibited higher normalized generality compared to AI systems and other animals.
- AI Specialization: Some AI systems, particularly those designed for specific tasks, showed high capability but lower generality, indicating specialization.
- No Consistent SLODR: In most scenarios analyzed, the correlation between capability and normalized generality was either positive or near zero, contradicting the traditional SLODR hypothesis. For example, in the Elithorn's mazes, the correlation was a mere 0.03.
- Quantitative Evaluation: The paper provides quantitative measures (e.g., normalized generality values ranging from -1 to +1) for various agents and tasks, allowing for direct comparison. For instance, in the Iris flower classification task using TDU difficulty, machines achieved a mean normalized generality of 0.84.
Limitations and Future Work
- Difficulty Metric Dependence: The analysis is sensitive to the chosen difficulty metric. The paper acknowledges that developing universal difficulty metrics for all tasks remains a significant challenge.
- Intrinsic vs. Extrinsic Difficulty: While intrinsic difficulty measures are preferred for their objectivity, extrinsic measures are sometimes necessary, introducing potential biases.
- Developmental and Potential Scenarios: The current framework primarily focuses on "consolidated" performance. Extending it to assess potential or developmental intelligence (i.e., the ability to learn) requires further research into how difficulty functions would adapt to learning processes and assistance levels.
- Universal Difficulty Measure: The authors suggest that developing a universal measure of difficulty for all tasks is a crucial area for future research.
- Revisiting Existing Data: The paper advocates for re-analyzing existing datasets from human, animal, and AI research using their generality analysis framework.
- Reframing Intelligence: The ultimate goal is to move beyond monolithic scores of intelligence and characterize individuals by their characteristic curves, with capability and generality as key indicators, offering a more nuanced understanding of "General Intelligence."
Summary updated