Exploiting Inductive Biases in Surrogate Benchmarks

This direction is no longer present in the current build. Its last version stays readable and citable here.

Version 1 of 123 papers25 conceptsfrozen 2026-10-04

In fields ranging from drug discovery to neural architecture search (NAS), the "ground truth" is often prohibitively expensive. Testing a new protein sequence might take weeks in a wet lab; training a single deep learning model to completion can cost thousands of dollars. To circumvent these bottlenecks, researchers use surrogate benchmarks: fast, cheap mathematical models trained to mimic the behavior of the real world.

However, a recurring pattern across recent research suggests that these surrogates are not neutral observers. Instead, they often create a "hallucination loop" where optimization algorithms appear to perform brilliantly by exploiting the specific inductive biases and artifacts of the surrogate model, rather than solving the underlying physical or logical problem.

The Circularity of Surrogate Evaluation

The fundamental appeal of a surrogate is its ability to provide an immediate "score" for any proposed design. In high-energy physics, Csaba Balázs, Melissa van Beekveld, and colleagues note in A comparison of optimisation algorithms for high-dimensional particle and astrophysics applications that navigating these "likelihood landscapes" is essentially a game of hide-and-seek.

The danger arises when the surrogate model is trained on data generated by a specific optimization algorithm. This creates a circularity: the surrogate learns to favor the search patterns of its parent algorithm. When a new optimizer is tested against this surrogate, it may be unfairly penalized simply because it explores the landscape differently, or unfairly rewarded if it mimics the original algorithm’s biases.

This problem is particularly acute in Neural Architecture Search (NAS). To avoid training thousands of models, researchers use surrogate benchmarks like those described by Arber Zela, Frank Hutter, and others in Surrogate NAS Benchmarks.

The "Inductive Bias Overlap" occurs when the search algorithm and the surrogate benchmark share the same underlying architecture (e.g., both are Graph Neural Networks). As Shen Yan and Colin White argue in NAS-Bench-x11 and the Power of Learning Curves, the search process can "game" the surrogate. The optimizer finds architectures that the surrogate thinks are good because they fit the surrogate's internal modeling assumptions, even if those architectures would perform poorly on real hardware.

The Hallucination Trap in Offline Optimization

In Offline Model-Based Optimization (MBO), we try to find a high-performing design (like a robot limb or a chemical formula) using only a static dataset, without any new experiments. To evaluate these methods, researchers use "oracles"—pre-trained simulators that act as the ground truth.

As Brandon Trabucco and Sergey Levine demonstrate with Design-Bench, these oracles are themselves approximations. Optimization algorithms often find "adversarial" designs—configurations that the surrogate oracle assigns a high score to, but which are physically impossible or biologically inactive.

Ye Yuan and Can Chen refer to this as "hallucination" in their work on Importance-aware Co-teaching. If the evaluator (the surrogate) has a blind spot, the optimizer will inevitably find it and pitch its tent there.

Inherited Biases in Foundation Models and Simulators

The shift toward using Foundation Models as evaluators introduces a new layer of this problem. In Foundation Model-oriented Robustness, Peiyan Zhang and Xing Xie show that using models like CLIP as "surrogate oracles" to test other models simply transfers the foundation model's own biases into the evaluation metric.

Similarly, in drug design, frameworks like MolChord by Wei Zhang and DrugR by Haoran Liu rely on automated docking simulations as reward signals. If the simulation doesn't perfectly capture the physics of a protein pocket, the AI agent will optimize for "simulated affinity," potentially generating molecules that are useless in a clinical setting. As Sixue Xing notes in the context of ClinicalReTrial, optimizing against a flawed simulator means the agent is merely "optimizing for false signals."

Breaking the Loop

To move past these self-referential evaluations, researchers are proposing several strategies:

  1. Bidirectional Learning: Can Chen and Yingxue Zhang propose BDI, which requires new designs to not only score well but also "explain" the existing training data, anchoring the model to reality.
  2. Information-Theoretic Rubrics: To stabilize LLM-as-a-judge systems, Wang Xu and colleagues developed a framework in Beyond Drift that uses Expected Information Gain to select evaluation questions, reducing the chance that the judge relies on its own internal heuristics.
  3. Multi-Fidelity Benchmarks: By providing full learning curves rather than single points, benchmarks like NAS-Bench-x11 allow for more robust testing that is harder for an optimizer to "game" with simple architectural tricks.

The core lesson of this research direction is a cautionary one: when we use a model to evaluate a model, we are no longer measuring performance against the world, but against a mirror. Understanding the "cracks" in that mirror—the inductive biases of the surrogate—is essential for true progress.

Papers in this version

History & citationVersion 1 · 2026-10-04

No longer present in the current build · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_exploiting_inductive_biases_in_surrogate_88a35be4,
  title = {Exploiting Inductive Biases in Surrogate Benchmarks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/exploiting-inductive-biases-in-surrogate-benchmarks/rdr_88a35be412b64e15a6f50fcb44308ea7},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_exploiting_inductive_biases_in_surrogate_88a35be4_v1,
  title = {Exploiting Inductive Biases in Surrogate Benchmarks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/exploiting-inductive-biases-in-surrogate-benchmarks/rdr_88a35be412b64e15a6f50fcb44308ea7/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}