Abstract
Neural Architecture Search (NAS) has significantly advanced automated model design for standard computer vision. However, its application to diverse domains—such as healthcare, genomics, and the physical sciences—is often hindered by rigid inductive biases, specifically the reliance on small convolutional kernels. We introduce DASH (Diverse-task Architecture SearcH), a novel NAS method that efficiently searches for optimal kernel sizes and dilations. By leveraging Fourier diagonalization and mixed-weight linearities, DASH decouples search complexity from kernel size, achieving a 10x speedup over traditional differentiable search methods. DASH establishes new state-of-the-art automated benchmarks on seven of ten tasks in the NAS-Bench-360 suite, frequently outperforming hand-designed expert architectures.
Introduction
While NAS has automated model design for natural images, it often fails in scientific and medical domains where data requires much larger receptive fields to capture long-range dependencies. Standard differentiable NAS methods, like DARTS, scale poorly as the search space of kernel sizes and dilations expands. This computational bottleneck forces a reliance on small kernels (e.g., $3 \times 3$), which are suboptimal for tasks like protein folding or PDE solving. DASH addresses this by enabling efficient search over massive receptive fields.
Methodology: The Fourier-Powered Search
DASH focuses on optimizing convolutional operations (kernel size $k$ and dilation $d$) within a fixed backbone topology. To make this search feasible, DASH introduces three primary innovations:
- Mixed-Weights: Instead of the standard approach of summing the outputs of multiple convolutions, DASH sums the kernels first. This reduces the number of expensive convolution operations per layer from $O(|K||D|)$ to $O(1)$.
- Fourier Convolution: By applying the Convolution Theorem, DASH performs element-wise multiplication in the frequency domain. This shifts computational complexity from $O(nk)$ to $O(n \log n)$, effectively making the search cost independent of kernel size.
- Kronecker Dilation: To further optimize, Kronecker products are used to construct dilated kernels on GPUs, providing an additional speedup over standard zero-padding.
Experimental Evaluation
DASH was evaluated using NAS-Bench-360, a benchmark designed to test generalization across ten diverse tasks, including Darcy Flow (PDE solving), PSICOV (protein folding), and NinaPro (sEMG signals).
Key Results
DASH demonstrated superior performance and efficiency across the board:
- State-of-the-Art Performance: DASH outperformed all other automated methods on 7 out of 10 tasks and surpassed hand-designed "expert" architectures on 7 tasks.
- Search Efficiency: The method achieved up to a 10x speedup in search time compared to naive differentiable search. On most tasks, the total search and retraining time was less than twice the time required to train a standard CNN backbone.
Discussion and Conclusion
The success of DASH highlights that kernel size and dilation are critical hyperparameters for non-vision tasks. For instance, in Darcy Flow, DASH identified optimal kernels as large as $9 \times 9$ with significant dilations—configurations typically ignored by standard NAS. By moving to the frequency domain, DASH acts as a "great equalizer," making the exploration of large receptive fields computationally trivial.
While DASH currently focuses on square 2D kernels, future work could extend this to rectangular convolutions or incorporate other operators like self-attention. Nonetheless, DASH represents a significant step toward universal, efficient architecture search for scientific and diverse data domains.
Summary updated