Beyond Fixed Search Spaces in Automated Design

951 concepts754 papers10 proposals

Automated optimization promises to liberate engineering and machine learning from the limits of human intuition. Whether searching for the perfect neural network architecture, a novel loss function, or an optimized physical circuit, we deploy algorithms to explore millions of possibilities. Yet, a fundamental paradox undermines this promise: the algorithms can only explore what we permit them to see.

Most automated discovery frameworks operate within a predefined search space—a rigid cage built out of human-designed templates, discrete menus of operations, and fixed parameter bounds. Consequently, instead of discovering fundamentally new paradigms, these systems often merely generate incremental mutations of existing ideas. To achieve true autonomous discovery, researchers are developing strategies to break out of these predefined boundaries.

The Bottleneck of Predefined Templates

Why do fixed search spaces limit discovery? To make an optimization problem computationally tractable, engineers typically restrict the search engine to rearranging a small set of known, trusted components. However, this structural steering inherently biases the system against radical innovation.

Consider physical design automation. In FALCON: An ML Framework for Fully Automated Layout-Constrained Analog Circuit Design, Asal Mehradfar et al. note that automated systems selecting from a fixed library of templates are unable to generate novel circuit architectures from scratch. The optimization process is confined to existing human-designed topologies. A similar bottleneck occurs in nanophotonic structure design; Jungtaek Kim et al. demonstrate in Multi-BOWS: multi-fidelity multi-objective Bayesian optimization with warm starts for nanophotonic structure design that methods relying on predefined structural representations cannot generate entirely novel geometric topologies because they are restricted to searching within fixed parameter bounds.

This "template tyranny" is equally restrictive in software and algorithmic design:

If an optimization algorithm is only allowed to choose from a discrete menu of human-designed options, it can never discover a category of solutions that falls outside that menu.

From Discrete Menus to Continuous Sculpting

To escape the constraints of a discrete menu, one powerful intuition is to relax the search space into a continuous landscape. Traditional Neural Architecture Search (NAS) requires a predefined list of discrete operations (e.g., choosing explicitly between a $3\times3$ convolution, a $5\times5$ convolution, or a max-pooling layer). This discrete framing creates a fragmented search space that is difficult to navigate efficiently without heavy human filtering.

In Differentiable Neural Network Architecture Search, Richard Shin et al. bypassed this by treating architecture selection as a continuous optimization problem, learning the network structure and its weights simultaneously within a single "super-network."

Taking this concept to its logical conclusion, David W. Romero introduced DNArch: Learning Convolutional Neural Architectures by Backpropagation. Instead of picking from a predefined menu of discrete operations, the author proposed treating the entire architecture—including depth, width, and kernel sizes—as continuous entities.

Discrete Search Space (Constrained Menu):
[3x3 Conv] ---> [5x5 Conv] ---> [Max Pool]  ==> Only combinations of these exist.

Continuous Search Space (Differentiable Sculpting):
[Continuous Weight & Size Mask] ------------> Sculpted smoothly via Backprop.

By "sculpting" the architecture using differentiable masks during a single training run, the framework optimizes structural variables without needing a predefined component list. This continuous relaxation allows the algorithm to discover highly unconventional, asymmetric network configurations that a human designer would never think to include in a discrete candidate pool.

Generative Discovery and Open-Ended Vocabularies

Another major strategy for breaking search space constraints replaces rigid structural skeletons with generative models. Instead of restricting an algorithm to mutating pre-existing code blocks or combining specific sub-functions, recent frameworks leverage Large Language Models (LLMs) and open-ended evolutionary loops to write code and propose hypotheses from scratch.

In AutoSGNN: Automatic Propagation Mechanism Discovery for Spectral Graph Neural Networks, Kai Wu et al. combined the creative coding capabilities of LLMs with iterative evolutionary strategies. This allowed the system to discover entirely custom propagation mechanisms, moving past the constraints of traditional black-box NAS frameworks that are restricted to predefined discrete components.

Similarly, in the domain of cognitive science, Milena Rmus et al. developed Generating Computational Cognitive models using Large Language Models. They noted that traditional automated hypothesis generation relies on predefined code templates, which inadvertently steer the search toward familiar architectures. By using LLMs to generate the code directly, their pipeline avoids this structural steering, discovering novel theoretical frameworks that frequently outperform decades of human-crafted models.

This shift from optimization to open-ended generation is summarized by Peter Jansen et al. in CodeScientist: End-to-End Semi-Automated Scientific Discovery with Code-based Experimentation. The authors emphasize that traditional autonomous discovery systems are bottlenecked because they focus on incremental mutations of existing code to improve benchmark scores. By moving to open-ended code generation, systems can explore fundamentally new tasks, metrics, and theoretical assumptions. When the search space is defined by the fluid vocabulary of language and logic rather than a fixed structural template, automated design systems finally gain the capacity to surprise us.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_predefined_search_space_constraints_92112125,
  title = {predefined search space constraints},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/predefined-search-space-constraints/rdr_921121250c7f45d591d3481afa95324c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_predefined_search_space_constraints_92112125_v1,
  title = {predefined search space constraints},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/predefined-search-space-constraints/rdr_921121250c7f45d591d3481afa95324c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake