TL;DR
Project Auto-World introduces an automated framework that leverages Large Language Models (LLMs) to benchmark and stress-test neural relational reasoning models. By using evolutionary algorithms and autonomous agentic search loops to discover graph-sampling code, the system exposes hidden blind spots in state-of-the-art models and establishes a self-improving pipeline for robust logical reasoning.
What Problem Does This Paper Address?
Neural networks frequently struggle with systematic generalization—the ability to apply learned logical regularities to problem instances that are more complex than those encountered during training. In relational reasoning (such as predicting links in a knowledge graph), evaluating this capability is notoriously difficult because it is rarely clear a priori what structural properties make a problem hard.
Traditional benchmarks rely on simple metrics like inference depth (the number of rule applications required to find an answer) or the count of "off-path" distracting edges. However, these metrics have significant blind spots. A model might achieve perfect scores on standard test splits while remaining fragile to unmeasured structural variations, creating a critical bottleneck for developing truly robust neural reasoners.
Methods
Instead of manually designing hard test cases, the authors turn the evaluation process into an adversarial game where an LLM acts as the benchmark generator. Given a world defined by logical rules (Datalog), the goal is to find a Python-based priority function that samples knowledge graphs capable of maximizing the failure rate of a target reasoning model (specifically, an Edge Transformer).
The paper explores two primary automated search strategies:
- LLM-Driven Evolutionary Search (FunSearch): The LLM acts as an evolutionary operator within a multi-island pool. It analyzes pairs of existing priority functions, proposes mutated/improved Python code, and receives a fitness score based on how effectively the generated graphs fool the evaluator.
- Autonomous Auto-Research Agent: A continuous, agent-in-the-loop procedure inspired by Andrej Karpathy's "auto-research" concept. A coding agent (Claude Opus 4.6) iteratively writes, executes, and refines a single priority program over multiple rounds, maintaining full experimental context and tool-use capabilities to systematically drive model accuracy down.
To close the loop, the authors use the discovered hard instances to adversarially train a upgraded model, dubbed SUPERET, to see if neural reasoners can be systematically hardened against these automated attacks.
Datasets
The framework was validated across two distinct environments:
- NoRA 1.1: A challenging benchmark focused on extended family relationships. Unlike simpler datasets, NoRA requires complex off-path reasoning where the answer depends on combining multiple relational paths and properties.
- Iron Coast: A completely novel domain featuring rules and constraints generated entirely by Claude Opus 4.6. This dataset was introduced to demonstrate the feasibility of a fully autonomous loop where the LLM invents both the world rules and the benchmark samplers from scratch.
Key Insights and Ideas
- Discovery of Inferred Off-Path Edges: The evolutionary samplers successfully identified a novel dimension of difficulty that human researchers had overlooked: inferred off-path edges. While existing metrics only counted explicit distractor edges, the LLM generated graphs where the model had to first infer a distracting relationship (e.g., a loop fact), which completely broke the Edge Transformer's reasoning chain. The authors formalized this as the Off-Path Entailed Edge Count (OPEEC).
- The Agent vs. The Hardened Model: Adversarial training successfully creates a much stronger model (SUPERET) that easily resists standard evolutionary search attacks. However, when pitted against the end-to-end Auto-Research Agent, the agent successfully adapts its coding strategy, discovering new combinations of multi-hop chains and connectivity bonuses that break even the hardened model.
Key Results
- Baseline Vulnerability: Standard Edge Transformers trained on baseline data saw their exact-match accuracy plummet to near zero (or between 0.01 and 0.15) when evaluated on test sets generated by the evolved priority functions.
- SUPERET Robustness: Training the model on an augmented dataset combining baseline data and LLM-sampled adversarial data (SUPERET) raised its resistance against evolutionary attacks significantly, maintaining an exact-match accuracy of around 0.70.
- Agentic Breakthrough: As shown in the auto-research timeline, the infinite agentic loop successfully drove the Edge Transformer's accuracy down from an initial 0.70 to a mere 0.05 by round 20 by continuously introducing complex structural features like "Extreme multi-hop" and "connectivity bonuses."
Limitations and Future Work
- Rule Syntax Restrictions: The current framework is strictly limited to positive Datalog rules. It does not yet support more expressive logical paradigms, such as non-monotonic reasoning (negation-as-failure) or disjunctive rules.
- Transferability Questions: The authors note that it remains an open question whether a priority function evolved to break a model in one specific world (e.g., NoRA) retains its adversarial effectiveness when transferred to an entirely different set of world rules.
- Architecture Scope: The study primarily evaluates Edge Transformers; future work is needed to verify if these exact difficulty signals generalize to other competitive neural relational reasoning architectures.
Summary updated