The Proxy Task Transferability Gap

82 concepts68 papers1 proposal

Automating the design of deep neural networks—whether through Neural Architecture Search (NAS), data augmentation optimization, or symbolic optimizer learning—is computationally brutal. Evaluating a single candidate architecture or training recipe typically requires training a model to convergence, which can take hours or days on modern hardware. When searching through millions of possible configurations, doing so directly on large-scale target tasks (such as ImageNet or billion-parameter language models) becomes completely intractable.

To bypass this bottleneck, the standard paradigm relies on proxy tasks. Instead of searching on the full target setup, algorithms evaluate candidates on a downscaled surrogate:

  • Smaller datasets (e.g., searching on CIFAR-10 instead of ImageNet)
  • Downscaled models (e.g., fewer layers, smaller hidden dimensions, or reduced channel counts)
  • Shorter training cycles (e.g., training for 5 epochs instead of 300)

The foundational assumption of this paradigm is simple: relative performance rankings are preserved. If Architecture A outperforms Architecture B on a proxy task, it should also outperform Architecture B on the final target task.

However, extensive research has revealed that this assumption is frequently false. This discrepancy is known as the proxy task transferability gap (or proxy-to-target transfer gap). When the proxy and target tasks diverge, the search algorithm optimizes for the specific quirks of the proxy, leading to architectures that underperform or fail completely when scaled up.

Why the Gap Exists: Dimensions of Misalignment

The transferability gap isn't just random noise; it is a systematic consequence of how neural networks scale. The optimal structural choices for a small, constrained model or dataset are fundamentally different from those of a large, complex one.

Data Scale and Resolution Bias

A network topology that excels at processing low-resolution images (e.g., 32x32 pixels in CIFAR-10) often fails when scaled to high-resolution inputs (e.g., 224x224 pixels in ImageNet). As Fang et al. note in their work on EAT-NAS, structural designs optimized for tiny images suffer from severe dataset bias when transferred to larger tasks. A specific receptive field size or downsampling rate that captures global context on a thumbnail will only capture local textures on a full-scale image. Similarly, Gasteiger et al. demonstrated in GemNet-OC that architectural features optimizing performance on small, homogeneous molecular datasets can become counterproductive when applied to large, diverse real-world distributions.

Model Capacity and Compute Regimes

Hardware constraints often force differentiable search methods to operate on downscaled proxy models with small hidden dimensions or fewer layers (Zhao et al.). However, architectural advantages do not scale linearly with compute budgets. Dollár et al. show in Fast and Accurate Model Scaling that designs optimized for low-compute regimes (such as mobile-sized models) do not scale effectively to high-compute environments, where activation memory and hardware-specific latencies dominate runtime dynamics. This non-linear scaling also distorts optimization choices: for example, bit-width sensitivity in mixed-precision networks is highly dependent on the specific dataset scale, causing configurations optimized on small proxies to degrade at scale.

Task and Optimization Complexity

Simple image classification proxies are fundamentally blind to the spatial and temporal complexities required for advanced tasks. As reviewed by Mohan et al. in Neural Architecture Search for Dense Prediction Tasks in Computer Vision, architectures that appear optimal on reduced datasets or short training cycles fail to maintain their advantages when subjected to high-resolution, full-scale dense prediction tasks like object detection or semantic segmentation. The same issue arises in visual tracking, where classification proxies fail to capture temporal dynamics (Marvasti-Zadeh et al.).

Empirical Evidence of Deceptive Proxies

Several large-scale empirical studies have quantified just how poorly proxy performance correlates with target performance.

In a comprehensive study, Panda et al. evaluated architecture transferability across massive datasets in NASTransfer. They discovered a stark disconnect: search algorithms that yielded identical performance on a small source dataset diverged significantly when scaled to large target datasets like ImageNet22K. Strikingly, they found that many sophisticated search algorithms failed to outperform simple random sampling from a well-designed search space when evaluated at scale, demonstrating that proxy tasks can actively mislead search heuristics.

Perhaps the most surprising finding comes from Schmidhuber et al. in Is it enough to optimize CNN architectures on ImageNet?. By testing hundreds of architectures across diverse datasets, the authors revealed that optimizing strictly for a single massive benchmark like ImageNet can lead to architectural choices that are completely sub-optimal—or even negatively correlated—with performance on specialized or smaller-scale target applications.

This ranking instability is also highly visible in benchmarks like NATS-Bench (Liu et al.), which document how optimal topologies shift dramatically across different data distributions and sizes. The gap is not unique to network topologies; it also plagues automated data augmentation and optimizer design. For instance, the authors of Adversarial AutoAugment and Meta Approach to Data Augmentation Optimization (Hataya et al.) show that data augmentation policies optimized on small proxies or shallow models fail to generalize to large-scale training regimes.

To overcome the limitations of proxy tasks, recent research focuses on three primary paradigms: direct proxyless search, elastic scaling, and domain-aware search.

The most straightforward way to eliminate the transferability gap is to eliminate the proxy task entirely. Cai et al. pioneered this approach with ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware. By utilizing path binarization, they reduced the GPU memory footprint of differentiable NAS to a constant level, allowing the search to run directly on the target large-scale dataset (ImageNet) and target hardware. This direct optimization ensures that the discovered models natively exploit the target task's characteristics and hardware latencies.

Rather than searching from scratch at scale, some frameworks use progressive or extrapolative methods to bridge the gap efficiently. Fang et al. proposed an elastic architecture transfer mechanism in EAT-NAS, which takes a high-performing architecture discovered on a small proxy dataset and dynamically fine-tunes its topology for the larger target task. In the domain of large language models, Acun et al. introduced Composer: A Search Framework for Hybrid Neural Architecture Design, a framework that searches for optimal hybrid Attention-MLP sequences at a tiny scale but utilizes strict composition rules that reliably correlate with and extrapolate to billions of parameters.

Another elegant solution is to explicitly account for the distribution shift between the proxy and target tasks during the search process itself. In Adapting Neural Architectures Between Domains, Yanxi Li introduced a domain adaptation framework for NAS. By minimizing the distribution distance between the proxy domain (e.g., CIFAR-10) and the target domain (e.g., ImageNet) within the super-network optimization loop, the search algorithm is forced to select structural components that are invariant to the domain shift, successfully bridging the generalization gap without incurring the massive cost of full-scale target training.

Key Foundational Work

A Flexible Approach to Automated RNN Architecture Generation (2018) — contributed 3 concept elements to this direction.
IRLAS: Inverse Reinforcement Learning for Architecture Search (2019) — contributed 2 concept elements to this direction.
PC-DARTS: Partial Channel Connections for Memory-Efficient Differentiable Architecture Search (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_proxy_task_transferability_gap_bc3b9ec7,
  title = {proxy task transferability gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/proxy-task-transferability-gap/rdr_bc3b9ec7a53e47bb8b46c9fa92deb5c4},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_proxy_task_transferability_gap_bc3b9ec7_v1,
  title = {proxy task transferability gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/proxy-task-transferability-gap/rdr_bc3b9ec7a53e47bb8b46c9fa92deb5c4/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake