How wide should each layer of a neural network be? Width search answers this by trying many per-layer channel allocations. The cost of that search grows with the number of layers, because every layer adds another free choice. Engineers who design airfoils, compressor blades and fluidic channels faced the same problem with dense meshes. Their answer was to describe the whole shape with a handful of control points on a smooth curve, such as a B-spline (a piecewise-polynomial curve pulled into shape by a few control points). The optimizer then moves only those few points. This note proposes doing the same for networks. The width profile is a curve over normalized depth, set by three to five control points. Total size is not searched: it is solved analytically from the compute budget. Search then explores only the curve's shape, so its dimension no longer depends on how deep the network is.
This matters for cost in three ways. A search space whose dimension is fixed and small makes simple black-box search sample-efficient. A curve defined on normalized depth can be found on a shallow, cheap network and then drawn at any target depth, much as a spline surface can be meshed at any resolution. And a weight-sharing supernet trained only on networks that lie on curves has far fewer subnetworks competing for the same weights. The bet is that most of the accuracy available from expensive per-layer width search lies on this low-dimensional family of curves, and can be reached at a fraction of the GPU-hours.
Core Claim
The profile is defined as follows. Let $t \in [0,1]$ be normalized depth, $c_1,\dots,c_K$ be $K$ control values, and $B_k(t)$ be the B-spline basis functions. The shape of the profile is
$$\tilde w(t) = \sum_{k=1}^{K} B_k(t)\, c_k, \qquad \sum_k c_k = 1,$$
and layer $l$ of an $L$-layer network gets width
$$w_l = \mathrm{round}_8\big(s \cdot \tilde w(l/(L-1))\big),$$
where $s$ is a single scale factor solved by bisection so the network meets a target FLOP budget, and $\mathrm{round}_8$ rounds to a hardware-friendly multiple of 8. The searched object is the normalized shape $(c_1,\dots,c_K)$, which has $K-1$ free dimensions whatever $L$ is.
The claim is that searching this shape gives three things relative to searching per-layer widths at equal FLOPs:
- Equal or better accuracy at much lower search cost. We expect better accuracy specifically when the search budget is small.
- Transfer across depth. Shapes found on shallow, briefly trained networks transfer to deeper target networks.
- Better supernet rankings. Supernets trained only on curve-generated subnetworks rank candidates more faithfully than supernets trained on the full per-layer space.
Relation to Prior Work
Three papers motivate the claim that smooth, low-dimensional profiles are where good networks live.
- Designing Network Design Spaces found that good widths follow a quantized linear ramp $u_j = w_0 + w_a j$. That ramp is monotone, and its parameters were refined by manual inspection of populations rather than searched under a budget. Our curve family contains the linear ramp as the case $K=2$ and adds curvature, including hourglass profiles.
- Vision Transformer Slimming learns independent soft masks per layer. It reports that good transformer profiles are non-monotone, with middle layers wider than deep ones. This is direct evidence that a linear ramp is too restrictive.
- Adaptive Width Neural Networks learns each layer's width independently through an importance ordering. It never ties widths across depth.
Two papers constrain the design.
- Locally Free Weight Sharing for Network Width Search shows that supernet interference is a real cost in width search. It fixes this by enlarging the space with free-channel zones and bins. We go the opposite way and shrink the space.
- NAS-Bench-Suite warns that conclusions drawn in one search space often fail in others. That is why we test on both CNNs and vision transformers.
The method itself comes from shape optimization:
- NURBS and B-spline parameterizations of airfoils and blades in Sun et al. and Joint optimization of aerodynamic and mechanical of a transonic compressor rotor.
- The low-resolution deformation cages wrapped around high-resolution meshes in Learning Tool Morphology for Contact-Rich Manipulation Tasks with Differentiable Simulation.
- The Bézier-parameterized boundaries in NeuralFluid, where low-dimensional parametric optimization converged where CMA-ES and PPO stalled.
Two ideas carry over from that work: separating a non-dimensional "shape" from a "size" (here, the budget-solved $s$), and the resolution independence of curve representations.
Research Program
The work has three parts.
Premise check on NATS-Bench. First we test whether the premise holds where ground truth exists. The NATS-Bench size space enumerates all 32,768 five-layer width allocations, with full training results on CIFAR-10, CIFAR-100 and ImageNet-16-120. We fit each allocation with curves of $K=2,3,4$ control points and record the fit residual. We then ask how many of the top 1% of architectures at each FLOP level lie within rounding distance of the curve family, compared with a random 1%. Five layers is a weak test of depth independence. It is, however, a cheap and exact test of whether good allocations are smooth.
Search in curve space. We then run search in curve space with deliberately simple optimizers: random search, regularized evolution and CMA-ES over the $K-1$ shape coordinates. We do this both on NATS-Bench (by table lookup) and on real training runs. If a simple optimizer works here, that is the point: the contribution is the space, not the optimizer.
Two cost-saving mechanisms. Finally, we test the two mechanisms that make search cheap.
- Depth transfer. We search shapes on ResNet-18-depth and DeiT-Tiny-depth proxies with short training schedules, then draw the shapes at ResNet-50 and DeiT-S depth.
- Curve-restricted supernet. We train a weight-sharing supernet whose sampled subnetworks are drawn only from curves, keeping the usual sandwich of smallest and largest members. We measure its ranking fidelity against standalone training.
Baselines
Naive reference. The do-nothing reference is the hand-designed default profile at the same FLOPs: standard ResNet-50 stage widths uniformly scaled, and the isotropic DeiT-S uniformly slimmed. Beating it is necessary but not interesting, since Designing Network Design Spaces and Vision Transformer Slimming already show non-uniform profiles win.
Strongest CNN width search. This is CafeNet from Locally Free Weight Sharing for Network Width Search. It reports 76.9% ImageNet Top-1 for ResNet-50 at 2G FLOPs, against 76.2% for TAS and 75.6% for AutoSlim. The file gives no CafeNet search-cost figure, so we will measure CafeNet's GPU-hours ourselves with the released settings. Our method is retrained with an identical final training recipe. "Beating it" means one of two things. We reach at least 76.9% at 2G FLOPs for less measured search compute. Or we exceed 76.9% at equal search compute.
Strongest transformer width search. This is ViT-Slim, which reports a 40% parameter and FLOP reduction on DeiT-S with up to 0.6% higher ImageNet-1K accuracy, using about 43 GPU hours of search. We match its FLOP target. Beating it means equal accuracy in clearly fewer than 43 GPU hours, measured on our hardware for both methods.
Linear-ramp ablation. The RegNet-style linear profile is the $K=2$ special case. Searched with the same optimizer and budget, it isolates whether curvature matters.
Per-layer search at equal sample budget. Regularized evolution over per-layer widths, the best black-box method in NAS-Bench-Suite with average rank 2.36, run with exactly the same number of evaluations. This isolates the value of the reduced dimension.
Supernet ranking baseline. On NATS-Bench we compare against a standard uniformly sampled supernet over all 32,768 allocations. No reported supernet Kendall-tau result on the NATS-Bench size space was found on Lacuna, so we will measure both supernets.
Why not just…?
Why not just use the RegNet linear ramp? It is the cheapest low-dimensional profile and has strong evidence behind it. But ViT-Slim's finding that good transformer profiles are non-monotone suggests a monotone ramp leaves accuracy on the table. The $K=2$ versus $K=3,4,5$ ablation settles this directly. If $K=2$ matches higher $K$ on both ResNet and DeiT, we will report that the curve adds nothing beyond RegNet except depth transfer.
Why not just add a smoothness penalty to per-layer soft masks? This is the most serious threat. ViT-Slim with a total-variation penalty across layers could produce smooth profiles with no new parameterization. The difference is that a penalty still searches a depth-dimensional space, offers no way to transfer across depth, and adds a penalty weight to tune. We will run penalized ViT-Slim, sweeping the penalty weight inside the same total compute budget. If it matches curve search in accuracy and cost at the target depth, the remaining claim narrows to depth transfer, and we will say so.
Why not just run regularized evolution on the per-layer grid? On tabular benchmarks it is nearly free and very strong. The question is what happens when each evaluation is a real training run. The equal-sample-budget comparison measures best-found accuracy as a function of the number of evaluations, for spaces of dimension $L$ versus $K-1$. If per-layer search catches up within a few dozen evaluations, the dimension argument fails.
Why not just prune a large trained model? Structured pruning does give a non-uniform profile. But AutoSlim, a slimming method, trails CafeNet at 2G FLOPs (75.6% vs 76.9%). Pruning also needs a full large-model training run first, and that run counts as search cost.
Proposed Experiments
Premise check (NATS-Bench). This is decisive and cheap. If the top 1% of NATS-Bench size-space allocations are no closer to the $K\le 4$ curve family than random allocations are, on all three datasets, the premise fails. We would stop before ImageNet.
Search efficiency (NATS-Bench). We plot best-found test accuracy against the number of table queries, for curve-space and per-layer search. The reported ceilings are about 94.4% on CIFAR-10 and about 47.3% on ImageNet-16-120.
Depth transfer (ImageNet). This is the second decisive check. A shape searched on ResNet-18-depth proxies, drawn at ResNet-50 depth at 2G FLOPs, should come within 0.3 points of the shape searched directly at ResNet-50 depth. It should also beat a per-layer profile found at ResNet-18 depth and linearly interpolated to ResNet-50 depth. If interpolating per-layer profiles transfers just as well, the resolution-independence argument is empty. We repeat the transfer from DeiT-Tiny depth to DeiT-S and DeiT-B depth.
Supernet fidelity (NATS-Bench). We report Kendall tau between supernet-inherited accuracy and true accuracy, evaluated on a common held-out set of curve-generated architectures. The prediction is that the curve-restricted supernet ranks better, because fewer subnetworks compete for its shared weights.
Reporting. Every result is reported as accuracy at fixed FLOPs against measured search GPU-hours, with three seeds for all ImageNet final trainings.
References
- Designing Network Design Spaces
- Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Space
- Adaptive Width Neural Networks
- Locally Free Weight Sharing for Network Width Search
- NAS-Bench-Suite: NAS Evaluation is (Now) Surprisingly Easy
- Evolutionary generative design of supercritical airfoils: an automated approach driven by small data
- Joint optimization of aerodynamic and mechanical of a transonic compressor rotor based on neural network algorithm
- Learning Tool Morphology for Contact-Rich Manipulation Tasks with Differentiable Simulation
- NeuralFluid: Nueral Fluidic System Design and Control with Differentiable Simulation
- NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size