Improving Differentiable Neural Architecture Search by Encouraging Transferability

Parth Sheth, Pengtao XieICLR 2023 · OpenReview

TL;DR

Differentiable Neural Architecture Search (NAS) often fails in practice because it finds "degenerate" architectures that over-rely on skip connections and generalize poorly to unseen data. This paper introduces TETLO, a tri-level optimization framework that forces the search process to prioritize architectures whose learned representations are highly transferable to an auxiliary model. By matching "quadruple relative similarities," the method produces more robust, high-performing neural networks across ImageNet and CIFAR benchmarks.

The Problem: Stability in Differentiable NAS

Neural Architecture Search (NAS) aims to automate the design of high-performance neural networks. Differentiable NAS (like the popular DARTS algorithm) is highly favored for its computational efficiency, as it treats the architecture as a continuous variable that can be optimized via standard gradient descent.

However, differentiable NAS suffers from a notorious "stability" problem. During the search phase, the algorithm often converges to "degenerate" architectures. These are models filled with an excessive number of skip connections that perform well on validation data during the search but fail miserably on unseen test data. Current fixes usually involve early stopping or hard-coding limits on skip connections. These are often "band-aid" solutions that fail to address the underlying lack of generalization.

The TETLO Framework

To solve this, the authors propose TETLO (Transferability-Encouraging Tri-Level Optimization). Instead of the standard bi-level optimization (optimizing weights vs. architecture), they introduce a third layer focused entirely on transferability.

The TETLO tri-level optimization framework.

The framework operates in three end-to-end stages:

  1. Stage I: Train the network weights of a "main model" while keeping the architecture fixed.
  2. Stage II: Train an "auxiliary model" by transferring knowledge from the main model.
  3. Stage III: Update the main model’s architecture by maximizing the auxiliary model’s validation performance.

The "Secret Sauce": Quadruple Relative Similarities (QRS)

To transfer knowledge effectively between the main and auxiliary models, the authors introduce QRS. While most knowledge distillation methods look at individual data points or pairs, QRS looks at four points at once. It asks: "Is the similarity between pair (A, B) greater than the similarity between pair (C, D)?"

By forcing the auxiliary model to match these higher-order relationships found by the main model, the architecture is incentivized to learn a more robust data manifold that isn't just "memorizing" the search set.

Key Insights

  • Transferability as a Proxy for Quality: The core insight is that a "good" architecture shouldn't just be good at its own task; it should learn representations that are useful enough to help another model learn. This acts as a powerful regularizer against architecture degeneration.
  • Beyond Pairwise Distillation: Moving to 4th-order relationships (QRS) captures the structural geometry of the data better than simple feature matching. This ensures the "knowledge" being transferred is high-quality and structural.
  • Natural Stability: Unlike previous methods that manually capped the number of skip connections, TETLO suppresses degeneration naturally. If an architecture is too reliant on skip connections, its representations won't transfer well to the auxiliary model, and the tri-level optimizer will naturally move away from it.

Key Results

The researchers evaluated TETLO across standard computer vision benchmarks (CIFAR-10, CIFAR-100, and ImageNet) and specialized search spaces (S1-S4) designed to measure NAS robustness. TETLO consistently outperformed state-of-the-art differentiable NAS baselines:

Comparison of TETLO against state-of-the-art methods on CIFAR-100.
  • Error Reduction: On CIFAR-100, TETLO-PRDARTS achieved a test error of 15.81%, a significant improvement over the baseline PR-DARTS (16.48%).
  • ImageNet Performance: On ImageNet (1.2M images), TETLO-PCDARTS reached a Top-1 error of 23.3%, outperforming the standard PC-DARTS (24.2%).
  • Degeneration Resistance: In a stress test across different search spaces, the baseline DARTS degenerated 5 to 9 times out of 10 runs. TETLO reduced this to only 1 or 2 times, proving it is vastly more stable.
  • Efficiency: Despite the tri-level complexity, the authors used one-step gradient approximations to keep search costs comparable to standard methods (e.g., ~4.0 GPU days for ImageNet-scale searches).

Limitations and Future Work

  • Task Similarity: The method assumes that the main and auxiliary tasks are similar enough for transfer to be meaningful. If the tasks were wildly different, the "transferability" signal might become noise.
  • Optimization Complexity: While the one-step approximation works efficiently, tri-level optimization is mathematically delicate. Future work could explore more stable or higher-order optimization solvers.
  • Broadening Scopes: The paper focuses heavily on image classification. Applying TETLO to more diverse domains like Natural Language Processing (NLP) or Reinforcement Learning (RL) remains an open and exciting area for exploration.

Summary updated

Abstract

Differentiable neural architecture search methods are increasingly popular due to their computational efficiency. However, these methods have unsatisfactory generalizability and stability. Their searched architectures are often degenerate with a dominant number of skip connections and perform unsatisfactorily on test data. Existing methods for solving this problem have a variety of limitations, such as cannot prevent the happening of architecture degeneration, being excessively restrictive in setting the number of skip connections, etc. To address these limitations, we propose a new approach for improving the generalizability and stability of differentiable NAS, by developing a transferability-encouraging tri-level optimization framework which improves the architecture of a main model by encouraging good transferability to an auxiliary model. Our framework involves three stages performed end-to-end: 1) train network weights of a main model; 2) transfer knowledge from the main model to an auxiliary model; 3) optimize the architecture of the main model by maximizing its transferability to the auxiliary model. We propose a new knowledge transfer approach based on matching quadruple relative similarities. Experiments on several datasets demonstrate the effectiveness of our method.
BibTeX
@inproceedings{sheth2023improving,
  author = {Parth Sheth and Pengtao Xie},
  title = {{Improving Differentiable Neural Architecture Search by Encouraging Transferability}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=Tl8OmiibP99},
}
Figure