When we evaluate human intelligence, we draw a sharp distinction between recognition and production. A student might successfully guess the answer to a multiple-choice geometry question by eliminating unrealistic options, but asking them to prove the underlying theorem on a blank sheet of paper provides a far deeper assessment of their understanding.
In artificial intelligence research, a parallel tension has emerged. Traditional benchmarks for abstract reasoning—such as Raven’s Progressive Matrices (RPMs) or visual analogies—have long relied on a discriminative selection format, where a model chooses the correct answer from a set of predefined options. However, a growing body of research suggests that this format creates a fundamental illusion of competence. Instead of inducing abstract rules, models often exploit statistical shortcuts, superficial patterns, or flaws in the distractor options to arrive at the correct choice.
To address this, researchers are shifting toward generative evaluation. By forcing models to synthesize solutions from scratch, construct novel problems, or perform forward logical deductions, this paradigm aims to isolate true fluid intelligence from superficial pattern matching.
The Multiple-Choice Shortcut
Why does discriminative evaluation fail to measure genuine reasoning? The core issue lies in the optimization landscape of multiple-choice tasks. When a model is trained or evaluated on a fixed set of candidates, it does not need to construct a logical thought process from first principles. Instead, it can succeed through a process of elimination or by identifying statistical artifacts within the distractors themselves.
As argued by Ian Davidson et al. in their work on fluid machine intelligence, multiple-choice benchmarks allow systems to bypass actual logic by exploiting these statistical shortcuts (Towards Fluid Machine Intelligence: Can We Make a Gifted AI?). This limitation is pervasive across modalities. Whether evaluating visual perception rules (PQA: Perceptual Question Answering) or multimodal analogical reasoning (VOILA: Evaluation of MLLMs For Perceptual Understanding and Analogical Reasoning), discriminative formats mask whether a model truly understands the underlying relational logic or is simply performing sophisticated pattern matching among the fixed choices.
De-Novo Synthesis: Painting the Answer
The most direct alternative to selection is de-novo synthesis: requiring a model to generate the missing component of a puzzle from a completely blank canvas.
Consider a Raven's Progressive Matrix, where a grid of geometric images follows an implicit rule (e.g., progressive rotation or shape addition), and the final cell is missing. In a generative paradigm, the model cannot look at an answer key. It must infer the rule and directly render or synthesize the missing image.
Discriminative Paradigm (Selection):
[ Context Panels ] ---> [ Model ] ---> Choose from 8 pre-defined options (Prone to shortcuts)
Generative Paradigm (Synthesis):
[ Context Panels ] ---> [ Model ] ---> Synthesize missing panel from scratch (Requires true rule induction)
To achieve this in visual domains, researchers have integrated deep generative architectures into reasoning pipelines. For instance, Niv Pekar et al. introduced a neural architecture that pairs Variational Autoencoders (VAEs) with a relation-aware perceptual critic to synthesize missing matrix images from scratch (Generating Correct Answers for Progressive Matrices Intelligence Tests). Similarly, Bin Li et al. utilized latent Gaussian Process priors to capture underlying concept-changing rules, allowing models to directly paint the missing cells of a matrix puzzle (Raven's Progressive Matrices Completion with Latent Gaussian Process Priors).
When a model must generate every pixel of a solution based purely on inferred rules, it cannot rely on distractor biases. Success in this regime provides a much more robust guarantee that the system has internalized the underlying causal mechanisms of the task.
Analysis-by-Synthesis and Internal Generation
Generative reasoning is not merely an evaluation tool; it can also be a mechanism for more robust problem-solving. Even when the final evaluation format requires selecting an answer, models can adopt an analysis-by-synthesis framework to protect against shortcut exploitation.
Instead of directly mapping the input puzzle to the multiple-choice candidates, an analysis-by-synthesis model first generates an internal prediction or mathematical representation of the solution. Chi Zhang et al. demonstrated this by developing a learner that abduces hidden rules and "imagines" the correct answer before matching it against the available options (Abstract Spatial-Temporal Reasoning via Probabilistic Abduction and Execution).
In a closely related approach, the authors showed that predicting an explicit algebraic representation of the transformation logic prior to selection significantly enhances a model's systematic generalization (Learning Algebraic Representation for Systematic Generalization in Abstract Reasoning). By decoupling the discovery of the rule from the evaluation of the candidates, the model mimics human-like conceptualization and avoids overfitting to dataset artifacts.
Expanding the Generative Frontier: From Test-Takers to Test-Makers
The philosophy of generative evaluation extends far beyond visual puzzles. It manifests in several distinct problem framings across language, logic, and code:
- Test-Making and Puzzle Generation: One of the highest expressions of understanding is the ability to construct a valid problem for others. Melissa Roemmele et al. shifted the evaluation focus from test-taking to test-making, demonstrating that an LLM's capacity to author valid commonsense reasoning items serves as a rigorous probe of its logical depth (From Test-Taking to Test-Making: Examining LLM Authoring of Commonsense Assessment Items). In the visual domain, Kalliopi Basioti et al. designed frameworks capable of generating entirely novel, rule-consistent visual puzzles from abstract specifications (GenVP: Generating Visual Puzzles with Contrastive Hierarchical VAEs).
- Forward Logical Deduction: Traditional automated theorem proving often works backward from a specific goal. However, Xiaohan Lin et al. argue that forward deduction—generating new, valid theorems and reusable lemmas from axioms—aligns more naturally with autoregressive language models and directly tests their ability to architect complex logical structures (ATG: Benchmarking Automated Theorem Generation for Generative Language Models).
- Hypothesis Space Exploration: Generative evaluation can also mean requiring a model to output the full spectrum of potential rules that fit a pattern. Antonia Wüst et al. leveraged this by evaluating whether a model could generate a diverse set of potential rules for abstract visual patterns, probing whether the correct concept exists anywhere within the model's internal hypothesis space (Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?).
The Complementary Value of Selection
While the research direction strongly favors generative tasks for robust evaluation, the relationship between generation and selection is nuanced. Generation is a high-entropy task; it requires not just conceptual understanding, but also flawless technical execution (such as rendering a precise image or writing syntactically perfect code).
When a model fails a generative task, it is often difficult to diagnose why. Did it fail because it lacked the abstract reasoning capability, or because it made a minor execution error?
To isolate these factors, researchers sometimes intentionally convert high-entropy generation tasks into controlled discriminative formats. For example, Donghyeon Shin et al. converted the highly challenging Abstract and Reasoning Corpus (ARC) into a multiple-choice format (From Generation to Selection: Findings of Converting Analogical Problem-Solving into Multiple-Choice Questions). They found that lowering the performance floor in this way helps isolate intermediate reasoning stages—like understanding and application—which might otherwise be masked by execution difficulties.
Similarly, in software engineering, Hong Yi Lin et al. reframed generative code refinement into a multi-step multiple-choice comprehension task (CodeReviewQA: The Code Review Comprehension Assessment for Large Language Models). This controlled context prevents the model from simply reciting memorized code from its training data, forcing it to actively evaluate specific alternatives.
Summary
The shift from discriminative selection to generative synthesis represents a maturation in how we evaluate machine intelligence. By closing the door on statistical shortcuts and candidate elimination strategies, generative evaluation forces models to demonstrate the active, first-principles construction of logical thought. Whether through painting missing pieces, generating counterfactual arguments, or authoring entirely new tests, true fluid intelligence is increasingly measured not by what a model can recognize, but by what it can create.
Key Foundational Work
Generating Correct Answers for Progressive Matrices Intelligence Tests (2020) — contributed 3 concept elements to this direction.
Towards Fluid Machine Intelligence: Can We Make a Gifted AI? (2019) — contributed 2 concept elements to this direction.
Learning Perceptual Inference by Contrasting (2019) — contributed 1 concept elements to this direction.