Automated optimization promises to liberate engineering and machine learning from the limits of human intuition. Whether searching for the perfect neural network architecture, a novel loss function, or an optimized physical circuit, we deploy algorithms to explore millions of possibilities. Yet, a fundamental paradox undermines this promise: the algorithms can only explore what we permit them to see.
Most automated discovery frameworks operate within a predefined search space—a rigid cage built out of human-designed templates, discrete menus of operations, and fixed parameter bounds. Consequently, instead of discovering fundamentally new paradigms, these systems often merely generate incremental mutations of existing ideas. To achieve true autonomous discovery, researchers are developing strategies to break out of these predefined boundaries.
The Bottleneck of Predefined Templates
Why do fixed search spaces limit discovery? To make an optimization problem computationally tractable, engineers typically restrict the search engine to rearranging a small set of known, trusted components. However, this structural steering inherently biases the system against radical innovation.
Consider physical design automation. In FALCON: An ML Framework for Fully Automated Layout-Constrained Analog Circuit Design, Asal Mehradfar et al. note that automated systems selecting from a fixed library of templates are unable to generate novel circuit architectures from scratch. The optimization process is confined to existing human-designed topologies. A similar bottleneck occurs in nanophotonic structure design; Jungtaek Kim et al. demonstrate in Multi-BOWS: multi-fidelity multi-objective Bayesian optimization with warm starts for nanophotonic structure design that methods relying on predefined structural representations cannot generate entirely novel geometric topologies because they are restricted to searching within fixed parameter bounds.
This "template tyranny" is equally restrictive in software and algorithmic design:
- Machine Learning Pipelines: In Automated Machine Learning for Remaining Useful Life Predictions, Marc-André Zöller et al. show that pipeline optimization is limited by a fixed search space of candidate architectures and hyperparameters, leaving the system unable to synthesize entirely new model structures.
- Loss Function Discovery: Frameworks like the one proposed by Hongyang Gu et al. in AutoLoss-GMS: Searching Generalized Margin-based Softmax Loss Function for Person Re-identification are restricted to predefined mathematical templates (such as margin-based softmax). As Peidong Liu et al. observe in Loss Function Discovery for Object Detection via Convergence-Simulation Driven Search, relying on a predefined set of mathematical operators selected by human experts excludes novel functional forms that require more complex, non-standard building blocks.
- Hardware-Software Co-Design: When jointly optimizing neural networks and hardware accelerators, Yonggan Fu et al. point out in Auto-NBA: Efficient and Effective Search Over the Joint Space of Networks, Bitwidths, and Accelerators that frameworks typically rely on predefined hardware templates (such as chunk-based pipelines). This dependency prevents the discovery of unconventional hardware designs. Mengquan Li et al. identify the same limitation in optical hardware accelerator search in O-HAS: Optical Hardware Accelerator Search for Boosting Both Acceleration Performance and Development Speed.
If an optimization algorithm is only allowed to choose from a discrete menu of human-designed options, it can never discover a category of solutions that falls outside that menu.
From Discrete Menus to Continuous Sculpting
To escape the constraints of a discrete menu, one powerful intuition is to relax the search space into a continuous landscape. Traditional Neural Architecture Search (NAS) requires a predefined list of discrete operations (e.g., choosing explicitly between a $3\times3$ convolution, a $5\times5$ convolution, or a max-pooling layer). This discrete framing creates a fragmented search space that is difficult to navigate efficiently without heavy human filtering.
In Differentiable Neural Network Architecture Search, Richard Shin et al. bypassed this by treating architecture selection as a continuous optimization problem, learning the network structure and its weights simultaneously within a single "super-network."
Taking this concept to its logical conclusion, David W. Romero introduced DNArch: Learning Convolutional Neural Architectures by Backpropagation. Instead of picking from a predefined menu of discrete operations, the author proposed treating the entire architecture—including depth, width, and kernel sizes—as continuous entities.
Discrete Search Space (Constrained Menu):
[3x3 Conv] ---> [5x5 Conv] ---> [Max Pool] ==> Only combinations of these exist.
Continuous Search Space (Differentiable Sculpting):
[Continuous Weight & Size Mask] ------------> Sculpted smoothly via Backprop.
By "sculpting" the architecture using differentiable masks during a single training run, the framework optimizes structural variables without needing a predefined component list. This continuous relaxation allows the algorithm to discover highly unconventional, asymmetric network configurations that a human designer would never think to include in a discrete candidate pool.
Generative Discovery and Open-Ended Vocabularies
Another major strategy for breaking search space constraints replaces rigid structural skeletons with generative models. Instead of restricting an algorithm to mutating pre-existing code blocks or combining specific sub-functions, recent frameworks leverage Large Language Models (LLMs) and open-ended evolutionary loops to write code and propose hypotheses from scratch.
In AutoSGNN: Automatic Propagation Mechanism Discovery for Spectral Graph Neural Networks, Kai Wu et al. combined the creative coding capabilities of LLMs with iterative evolutionary strategies. This allowed the system to discover entirely custom propagation mechanisms, moving past the constraints of traditional black-box NAS frameworks that are restricted to predefined discrete components.
Similarly, in the domain of cognitive science, Milena Rmus et al. developed Generating Computational Cognitive models using Large Language Models. They noted that traditional automated hypothesis generation relies on predefined code templates, which inadvertently steer the search toward familiar architectures. By using LLMs to generate the code directly, their pipeline avoids this structural steering, discovering novel theoretical frameworks that frequently outperform decades of human-crafted models.
This shift from optimization to open-ended generation is summarized by Peter Jansen et al. in CodeScientist: End-to-End Semi-Automated Scientific Discovery with Code-based Experimentation. The authors emphasize that traditional autonomous discovery systems are bottlenecked because they focus on incremental mutations of existing code to improve benchmark scores. By moving to open-ended code generation, systems can explore fundamentally new tasks, metrics, and theoretical assumptions. When the search space is defined by the fluid vocabulary of language and logic rather than a fixed structural template, automated design systems finally gain the capacity to surprise us.