One Programming, Many Budgets: Training Weight-Shared Networks for Context-Dependent Analog Error

Analog in-memory computing stores a neural network's weights as electrical conductances in a grid of memory cells, called a crossbar. Applying input voltages to the rows lets the column currents compute a matrix-vector product directly, following Ohm's and Kirchhoff's laws. Reprogramming these cells is slow and wears them out. A chip that serves several energy or latency budgets, running a thin network when the battery is low and a wide one when it is not, should therefore store one weight-shared "supernet" and switch between its sub-networks by turning rows and columns on or off. This note asks what happens to the shared cells when that is done. Our claim is that in analog hardware the error a stored weight contributes is not a fixed property of the cell. It depends on which sub-network is running. Wire resistance drops voltage along each line in proportion to the total current, so it drops less when fewer rows are active. The converter that digitizes column currents was sized for some current range, and narrow sub-networks produce much smaller currents, so they use only part of that range. The same programmed conductance therefore produces different effective weights in different sub-networks.

Current hardware-aware training injects one context-free noise model into every forward pass. When it is applied to a supernet, it optimizes the shared weights against an error distribution that no deployed sub-network actually sees. We propose subnet-conditioned hardware-aware training, in which the simulated error for each sampled sub-network is computed from that sub-network's own active rows, columns and currents. We also give the measurement protocol needed to decide whether this matters: first how large the context dependence is, then whether cheap per-budget calibration already removes it, and then whether conditioning improves worst-budget accuracy after months of simulated conductance drift.

Core Claim

When a weight-shared network is programmed once onto an analog crossbar and executed at multiple widths, the shared weights face a family of sub-network-specific error distributions. A single noise model cannot stand in for that family. The mismatch is a simulation-to-silicon gap that arises only because of weight sharing. The error of a shared cell has two parts: a component that is the same in every sub-network, such as programming noise and drift, and a component that changes with the active mask, such as line-resistance voltage drop and converter range use. We claim this second component is large enough to reorder which budgets fail after drift. Training against it, rather than against an averaged noise model, should raise the accuracy of the worst budget by more than per-budget recalibration can. The objective is

$$\min_{W}\; \mathbb{E}_{a\sim p(a)}\,\mathbb{E}_{\xi\sim \mathcal{N}}\Big[\mathcal{L}\big(f_a(x;\,h(W,\xi,m_a,x))\big)\Big],$$

with the following terms. $W$ is the shared high-precision "shadow" weight. $a$ is a sub-network drawn from sampling distribution $p(a)$, and $m_a$ is its row and column mask. $\xi$ is a draw of the context-independent device noise (programming noise, read noise, drift). $h$ maps these, together with the current input $x$, to the effective weights and output quantization actually experienced on the crossbar. $\mathcal{L}$ is the task loss of sub-network $f_a$. Standard hardware-aware training is the special case where $h$ ignores $m_a$ and $x$.

Where This Sits in Prior Work

Our clearest motivation comes from analog work showing that the relevant non-idealities are input- and layout-dependent. AIM shows on a post-layout 7nm processing-in-memory chip that voltage drop depends on the toggle rate and bit patterns of the weights. It reduces the drop from 140mV to about 43mV through software-hardware co-design, and notes that analog designs are the more sensitive. AIM treats one fixed model, however, so which weights are active never changes. Rasch et al. report that noise at the output, such as converter noise, hurts accuracy more than noise in the weights. That points us toward converter range use as the second subnet-dependent term. DWP shows that robustness degrades when the noise level assumed in training departs from deployment conditions. This is exactly the failure a context-free noise model creates for every sub-network except the one it was calibrated on. It also constrains our design: we must not simply add more noise.

The training method we build on is the shadow-weight hardware-aware training loop in AIHWKit and Rasch et al.. In that loop, noise is applied to the weights in the forward pass and exact gradients update a clean digital copy. The kit supplies the phase-change memory device models, drift, converter models and line-resistance models we use. The drift behavior we evaluate against follows the power law documented by Spoon et al.. All of this work trains single static models.

The closest intersection is XPert. It co-searches channel widths, column sharing and converter precision with a differentiable supernet under simulated noise. XPert uses the supernet as a search device, extracts one design per chip, and applies context-free noise. It does not ask what happens when one programmed array must serve many sub-networks, and it does not analyze interference between sub-networks. UH-NAS finds that cheap architecture-quality proxies become negatively correlated with accuracy under physical non-idealities. We take this as a warning that rankings measured in simplified simulation can invert. UH-NAS trains every candidate separately, so weight sharing never arises.

On the supernet side, Xu et al. tie gradient conflict between sub-networks to how far apart they are in the architecture space. Shipard et al. argue that the apparent interference is mostly sampling bias and that uniform epoch-level sampling removes it. CaLR with momentum separation adjusts learning rates and momentum buffers per sub-network complexity. CompOFA restricts the space to 243 coupled depth-width models. FlexRank enforces nested low-rank structure so budgets do not compete. All five work in ideal floating point, where a shared weight means the same number in every sub-network. Our setting breaks that assumption. It also gives a clean test of the Xu–Shipard disagreement: if interference is only sampling bias, uniform sampling should suffice even when hardware makes the sub-networks' error landscapes differ.

Research Program

The first step is measurement. We map width-slimmable ResNet and MobileNet-style networks in the CompOFA compound space onto AIHWKit phase-change memory tiles, with programming noise, drift, converters and line resistance enabled. We then quantify the context-dependent error term. For each shared cell, we compare the variance of its effective weight across sub-networks, at fixed device noise, with its variance across device-noise draws within one sub-network. We do the same for column-level converter range use. The ratio of these two variances is the quantity the whole proposal rests on. We report it as a function of array size and line resistance, because it should grow as arrays grow and as wiring becomes more resistive.

The second step is a fast conditioned error model. Solving the full resistive network in every training step costs cubic time per layer and is too slow. We therefore fit a first-order surrogate. The attenuation of each cell is written as a function of the cumulative active current upstream along its row and column, computed from $m_a$ and the batch activations. Converter clipping and step size are computed from the actual column-current range of sub-network $a$. The surrogate is checked against full line-resistance solves on held-out layers. The full solver is kept only for evaluation, so the training-time surrogate never grades itself.

The third step is conditioned supernet training. Sub-networks are sampled uniformly at the epoch level, following Shipard et al., together with sandwich sampling of the smallest and largest widths. Each sample's forward pass uses $h$ conditioned on its own mask. In the resulting weight-shared network, every budget, running from one programming of the array, is accurate at time zero and after simulated drift. As a diagnostic, we record cosine similarity between gradients of large and small sub-networks under clean, context-free-noise and conditioned-noise training. This tells us whether hardware noise evens out the sub-networks' losses or sharpens their conflict.

Baselines

The do-nothing reference is a digitally trained slimmable supernet programmed directly onto the simulated crossbar and evaluated at each width. It shows how much any hardware awareness buys. Rasch et al. report the analogous single-model effect: WideResNet-16 went from 25% error under direct mapping to a baseline-level 20% after hardware-aware training.

The strongest existing method is context-free hardware-aware supernet training. This is the AIHWKit noise model applied identically to every sampled sub-network, with the same uniform and sandwich sampling, plus the per-sub-network batch-norm recalibration that Once-For-All pipelines already perform. It is essentially what XPert's noisy supernet and Rasch et al.'s training would produce if combined. XPert reports 92.46% on CIFAR-10 and 56.7% on TinyImageNet under noise for its searched single designs. Neither number measures a multi-budget array programmed once, and no reported result was found on Lacuna for runtime width switching on a single programmed analog array. We therefore reproduce this baseline ourselves.

We add a stronger variant of it: context-free training plus per-budget calibration of the converter gain and a per-column affine output correction, fitted on a few hundred calibration images. This is what a hardware engineer would try first. We also apply CaLR with momentum separation to the context-free baseline. That paper reports Kendall's $\tau$ rising from 0.751 to 0.814 on NAS-Bench-201 CIFAR-10 over plain single-path training. Including it tests whether ordinary interference mitigation already absorbs the effect.

The upper reference is one independently hardware-aware-trained model per budget, each on its own array. It measures the price of sharing memory.

Two ablations isolate the key ingredient. In the first, the mask passed to $h$ is replaced by a random other sub-network's mask. This keeps the same noise magnitude and variety but removes the correct context, so it separates "right conditioning" from "more diverse noise," which is DWP's mechanism. In the second, the surrogate is replaced by the full solver on a small model, to measure what the approximation costs.

All methods share the data, the architecture space, the training epochs and wall-clock budget, the device configuration, three seeds, and an equal hyperparameter search. Conditioned training counts as beating the strongest baseline if two things hold for context-free training plus calibration. It must improve worst-budget accuracy after one simulated year of drift by more than two seed standard deviations. It must also close at least half of that baseline's gap to the independently trained models.

Why not just…?

Why not just recalibrate each budget after training? Per-budget converter gain and affine column correction cancel the average attenuation a sub-network sees. Line-resistance drop, however, depends on each input pattern, and AIM shows it tracks toggle rates and bit patterns rather than being a static offset. A per-column constant cannot follow that. This is the most serious alternative, and it is included as a baseline rather than argued away. If calibration closes the gap, our result becomes a negative finding: context dependence exists but can be fixed by calibration. We would report it that way.

Why not just fine-tune each extracted sub-network with hardware-aware training? That is the normal Once-For-All route. It produces separate weights per budget, which requires either reprogramming the array whenever the budget changes or storing several copies of the weights. Either option defeats the purpose of one programmed array. It remains the right upper reference, and it appears as such in the baselines.

Why not just keep the analog weights frozen and add a small digital adapter per budget? AHWA-LoRA keeps F1 within 1% on SQuAD and recovers over 13 F1 points after ten years of drift. HaLoRA cuts variance under stuck-at faults from 13.22 to 1.86 standard deviation. These results make per-budget digital adapters a real threat. The adapters can absorb low-rank, input-independent corrections. We test them directly, with adapters on top of both context-free and conditioned supernets at matched parameter counts. If they erase the difference, the practical recommendation becomes "context-free supernet plus per-budget adapters." If conditioning and adapters add up, both belong in the pipeline.

Why not just train with larger or more diverse noise? Broad noise pushes weights toward flat regions of the loss, as DWP does. DWP's own mismatch results show, however, that miscalibrated noise costs robustness, and DWP adds 3.2× training time. The wrong-mask ablation keeps noise diversity fixed while removing correct conditioning, so it answers this question directly.

Decisive Experiments

The benchmarks are CIFAR-10, CIFAR-100 and TinyImageNet, matching XPert's reporting, on the CompOFA 243-model width-depth space. If time allows, we add a nested low-rank MobileBERT on SQuAD v1.1 in the FlexRank style, which ties the work to AHWA-LoRA's setting. Evaluation always uses the full line-resistance solver and AIHWKit drift at one hour, one day, one month and one year. Metrics are per-budget and worst-budget accuracy at each drift time, the gap to independently trained models, and Kendall's $\tau$ between the supernet's accuracy ranking of budgets and the ground-truth ranking under full simulation. Where the kit's cloud hardware access permits, we will run a small check on physical hardware. We claim nothing about silicon until that check is done.

Three results decide the proposal. The first gate is the variance ratio from the measurement step. If the context-dependent term is small compared with programming noise at realistic array sizes, we stop and report that context-free models are adequate for weight-shared analog deployment. The second is whether calibration and adapter baselines close the gap. The third is whether conditioned training changes which budgets survive drift, meaning the drift-time ranking of budgets under context-free training disagrees with the ground truth while under conditioned training it agrees. The gradient-cosine diagnostic runs alongside these. It gives the first evidence on whether hardware noise supports Shipard et al.'s view that uniform sampling suffices, or Xu et al.'s view that conflict between distant sub-networks is intrinsic, in a setting where sub-networks truly differ in the error they experience.