Neural Architecture Search (NAS) has revolutionized model design by automating the discovery of high-performing neural networks. However, as noted by Thomas Elsken et al. in Neural Architecture Search: A Survey, the field has historically suffered from a profound bias toward image classification benchmarks.
Early automated design frameworks, such as those introduced by Masanori Suganuma et al. in A Genetic Programming Approach to Designing Convolutional Neural Network Architectures and Wei Wu et al. in Practical Block-Wise Neural Network Architecture Generation, proved that evolutionary algorithms and reinforcement learning could successfully discover modular building blocks. Yet, these blocks were optimized under a narrow paradigm: mapping an entire image to a single categorical label. When these classification-optimized structures are transferred to dense prediction tasks—such as semantic segmentation, object detection, or depth estimation—their performance frequently collapses.
To bridge this gap, a major research direction has emerged focusing on cross-task architectural generalization, aiming to design search spaces and optimization strategies that allow automated architectures to scale across diverse visual domains.
The Architectural Generalization Gap
Why do architectures optimized for classification fail when applied to downstream spatial tasks? The root cause lies in a fundamental structural tension between semantic abstraction and spatial localization.
Image classification rewards networks that progressively discard spatial resolution. By downsampling feature maps through successive pooling or strided convolutions, a network forces its layers to ignore where an object is and focus entirely on what it is.
Conversely, dense prediction tasks require the exact opposite behavior: they demand precise pixel-level localization. An architecture must retain high-resolution spatial details while simultaneously capturing deep semantic context.
Classification Search Space:
Input ──> [Downsample] ──> [Extract Semantics] ──> Global Label
Dense Prediction Search Space (Cross-Task NAS):
Input ──┬──> [High-Res Path] ───────┬──> Pixel-Level Output
└──> [Low-Res Semantic Path] ┘ (Multi-Scale Routing)
To systematically quantify this discrepancy, Yawen Duan et al. developed TransNAS-Bench-101: Improving Transferability and Generalizability of Cross-Task Neural Architecture Search. By evaluating thousands of network topologies across seven distinct visual tasks, the authors empirically demonstrated the existence of an architectural generalization gap: a structural cell topology that excels at classification is rarely optimal for tasks requiring complex spatial reasoning.
Designing Search Spaces for Spatial Reasoning
To enable cross-task generalization, researchers have shifted their focus from searching for rigid, sequential blocks to designing flexible, multi-scale search spaces.
Instead of forcing a network to follow a linear downsampling path, modern cross-task frameworks search for optimal feature routing across multiple resolutions simultaneously. For example, Liang-Chieh Chen et al. introduced the Dense Prediction Cell in Searching for Efficient Multi-Scale Architectures for Dense Image Prediction. This approach explicitly searches for multi-scale feature aggregations, allowing the network to dynamically capture objects at various scales without sacrificing spatial resolution.
To provide even greater structural flexibility, Peiwen Lin et al. proposed Graph-Guided Architecture Search for Real-Time Semantic Segmentation. By modeling the search space as a graph, the authors allowed unique cells to communicate across different resolutions on the fly. Similarly, Wuyang Chen proposed a multi-resolution search space in FasterSeg: Searching for Faster Real-time Semantic Segmentation, which leverages a co-searching teacher-student distillation process to preserve high-frequency spatial details while maintaining real-time inference speeds.
Scalable Transfer and Multi-Objective Generation
Executing a massive neural architecture search from scratch for every new task is computationally prohibitive. Therefore, achieving cross-task generalization requires scalable transfer mechanisms that can adapt existing structural knowledge to new domains efficiently.
One prominent strategy is structural fine-tuning and elastic seeding. Jiemin Fang and Yukang Chen developed EAT-NAS: elastic architecture transfer for accelerating large-scale neural architecture search, which uses a high-performing architecture discovered on a small source dataset as a "seed" and automatically fine-tunes its macro-structure for a larger target task. Taking this a step further, Sharath Nittur Sridhar and Souvik Kundu proposed InstaTune: Instantaneous Neural Architecture Search During Fine-Tuning. InstaTune bypasses the expensive pre-training search phase entirely by making off-the-shelf pre-trained models "elastic" during task-specific fine-tuning, allowing hardware-optimized sub-networks to be discovered instantaneously.
Ultimately, the goal of this research direction is to build universal generators capable of outputting task-tailored networks on demand. In POMONAG: Pareto-Optimal Many-Objective Neural Architecture Generator, Eugenio Lomurno et al. leveraged a many-objective diffusion process to generate Pareto-optimal architectures across 15 different datasets. By integrating multiple performance predictors directly into the generation phase, the framework balances accuracy, model size, and hardware constraints across diverse visual tasks simultaneously.
Why It Works: Decoupling Primitives from Routing
The core intuition behind cross-task architectural generalization relies on a crucial realization: the primitive operations of a neural network (such as convolutions, pooling, or self-attention) are largely task-agnostic, but the routing topology between those primitives is highly task-specific.
Traditional NAS treated the entire network as a homogeneous sequence of operations, assuming that if a block was good at extracting features for classification, it would be equally good at extracting features for segmentation. Cross-task NAS succeeds because it decouples these two concerns. It allows the micro-topology (the individual operations) to remain stable while optimizing the macro-topology (how features are combined, downsampled, and upsampled across scales).
By focusing the search on the multi-scale routing paths, cross-task NAS enables a single underlying optimization framework to dynamically allocate representation capacity where the target task demands it: globally and abstractly for classification, or locally and multi-scally for dense spatial prediction.
Key Foundational Work
Bayesian Meta-network Architecture Learning (2018) — contributed 1 concept elements to this direction.
Practical Block-Wise Neural Network Architecture Generation (2018) — contributed 1 concept elements to this direction.
Searching for Efficient Multi-Scale Architectures for Dense Image Prediction (2018) — contributed 1 concept elements to this direction.