Abstract
Differentiable architecture search (DAS) is a widely researched tool for the discovery of novel architectures, due to its promising results for image classification. The main benefit of DAS is the effectiveness achieved through the weight-sharing one-shot paradigm, which allows efficient architecture search. In this work, we investigate DAS in a systematic case study of inverse problems, which allows us to analyze these potential benefits in a controlled manner. We demonstrate that the success of DAS can be extended from image classification to signal reconstruction, in principle. However, our experiments also expose three fundamental difficulties in the evaluation of DAS-based methods in inverse problems: First, the results show a large variance in all test cases. Second, the final performance is strongly dependent on the hyperparameters of the optimizer. And third, the performance of the weight-sharing architecture used during training does not reflect the final performance of the found architecture well. While the results on image reconstruction confirm the potential of the DAS paradigm, they challenge the common understanding of DAS as a one-shot method.
Introduction
Neural Architecture Search (NAS) traditionally requires training thousands of candidate models, a process that is computationally expensive. Differentiable Architecture Search (DAS), popularized by the DARTS algorithm, promised a shortcut: treating the architecture itself as a continuous variable and optimizing it alongside network weights in a single "one-shot" pass. While DAS has succeeded in image classification, this study investigates its robustness when applied to inverse problems, such as signal reconstruction and deblurring.
Methods and Search Spaces
The researchers applied DAS to 1D signal reconstruction tasks using two distinct search spaces to evaluate the "one-shot" claim:
- Sequential Search Space: A linear sequence of 10 nodes where the model selects between "benign" operations (e.g., learnable gradient descent, CNNs) and "harmful" ones (e.g., noise, signal rolling).
- Non-Sequential Search Space: A complex, cell-based Directed Acyclic Graph (DAG) structure, mirroring the original DARTS setup.
To ensure statistical rigor, the authors performed 75 to 100 trials per configuration. They utilized synthetic 1D Cosine Wave data subjected to Gaussian blur and subsampling, allowing for the thousands of experiments necessary to analyze the variance and hyperparameter sensitivity of the DAS process.
The "One-Shot" Reality Check
The study reveals several critical challenges that undermine the standard perception of DAS as a reliable one-shot tool:
- The Correlation Gap: There is a "loose" correlation between the performance of the weight-sharing supernet during search and the performance of the final discretized architecture after retraining. A high-performing search phase does not guarantee a high-performing final model.
- High Instability: DAS results exhibit massive variance. Depending on the random seed or initialization, the same search space can yield either a top-tier architecture or a total failure.
- Hyperparameter Sensitivity: The success of DAS is heavily dependent on the hyperparameters of the optimizer (e.g., learning rates for architecture parameters vs. weights). Hyperparameters optimized for deblurring rarely transferred effectively to downsampling tasks.
Key Results
Quantitative analysis shows that while DAS can find novel hybrid architectures, it often struggles against simpler baselines. In search spaces containing only "good" operations, a simple random search often outperformed DAS. Furthermore, the inclusion of "harmful" operations significantly degraded DAS performance.
As seen in the results, when "harmful" operations were added to the blurred data task, the median PSNR for DAS dropped from 21.60 to 18.57. This suggests that while DAS can filter out bad operations, it is not consistently reliable in doing so compared to a robust random search.
Conclusion
The findings challenge the common understanding of DAS as a one-shot method. Due to high variance and hyperparameter sensitivity, achieving a high-quality result typically requires running DAS multiple times and selecting the best outcome—effectively turning it into a "multi-shot" method. For DAS to become a robust tool in new domains, future work must focus on better initialization strategies and the development of evaluation metrics that account for variance rather than just peak performance.
Summary updated