Abstract
Differentiable architecture search (DAS) is a widely researched tool for the discovery of novel architectures, due to its promising results for image classification. The main benefit of DAS is the effectiveness achieved through the weight-sharing one-shot paradigm, which allows efficient architecture search. In this work, we investigate DAS in a systematic case study of inverse problems, which allows us to analyze these potential benefits in a controlled manner. We demonstrate that the success of DAS can be extended from image classification to signal reconstruction, in principle. However, our experiments also expose three fundamental difficulties in the evaluation of DAS-based methods in inverse problems: First, the results show a large variance in all test cases. Second, the final performance is strongly dependent on the hyperparameters of the optimizer. And third, the performance of the weight-sharing architecture used during training does not reflect the final performance of the found architecture well. While the results on image reconstruction confirm the potential of the DAS paradigm, they challenge the common understanding of DAS as a one-shot method.
Introduction
Differentiable Architecture Search (DAS), specifically the DARTS algorithm, has gained significant traction in AutoML by framing architecture selection as a continuous optimization problem. Marketed as a "one-shot" solution, it promises to find optimal neural architectures in a single training run by optimizing a "supernet." While successful in image classification, its applicability to inverse problems—such as signal deblurring and reconstruction—remains under-explored. This study critically evaluates whether the one-shot promise holds for 1D inverse problems, revealing significant challenges regarding stability and hyperparameter sensitivity.
Experimental Methodology
The authors designed a controlled framework using two distinct search spaces to evaluate DAS. The first is a Sequential Search Space, a 10-layer linear chain. The second is a Non-Sequential Search Space, utilizing a cell-based directed acyclic graph (DAG) similar to the original DARTS implementation.
To test robustness, the search spaces were populated with both "benign" operations (e.g., CNNs, learnable gradient descent layers) and "harmful" operations (e.g., Gaussian noise or random shifts). A successful NAS algorithm must distinguish between these to find a high-performing hybrid model. The study employed 1D synthetic signals (cosine waves) corrupted by Gaussian blur and downsampling, measuring performance via Peak Signal-to-Noise Ratio (PSNR).
Key Findings
The investigation revealed three primary obstacles to the "one-shot" paradigm in inverse problems:
- The Correlation Gap: There is a troubling lack of correlation between the performance of the weight-sharing supernet during search and the final performance of the architecture after retraining. In non-sequential spaces, this correlation was occasionally negative, meaning the "best" model according to the search was actually one of the worst after retraining.
- Hyperparameter Sensitivity: DAS performance is highly dependent on the optimizer's hyperparameters, such as learning rates and weight decays for the architecture parameters ($\alpha$). Optimal settings for deblurring did not generalize to downsampling tasks.
- High Variance and Instability: Even with optimized hyperparameters, different random seeds led to a wide spread in final PSNR values. Because weights are initialized randomly, a "good" operation might perform poorly at the start, leading DAS to discard it prematurely.
Results Analysis
Despite these challenges, DAS demonstrated clear utility when the search space contained harmful operations, significantly outperforming random search.
As shown in the results, DAS achieved a median PSNR of 18.57 for the blur task in the "all" operations category, compared to just 8.10 for a random architecture. The search often converged on "hybrid" architectures that effectively combined unrolled optimization with convolutional layers, outperforming models designed purely with one approach.
Conclusion
The study concludes that while DAS can discover superior hybrid architectures for inverse problems, it is not a true "one-shot" method. Its success requires extensive hyperparameter tuning and statistical validation across multiple runs. Rather than a standalone solution, DAS should be viewed as a powerful but sensitive component within a larger architectural discovery pipeline that requires its own statistical evaluation.
Summary updated