Analog in-memory computing performs a neural network's matrix multiplications directly inside arrays of memory devices. The devices are resistive cells arranged in a grid called a crossbar, and the multiplication falls out of Ohm's and Kirchhoff's laws. This saves large amounts of energy, but the arithmetic is noisy. Device conductances drift over time, programming is stochastic, and the analog-to-digital converters that read the results quantize them coarsely. Engineers decide how much of this noise is acceptable using objective metrics such as classification accuracy, word error rate, or sample-level signal error. For generative audio those metrics are known to be poor judges. A drum loop can shift by a fraction of a millisecond and lose badly on signal-to-distortion ratio while sounding identical to a listener.
This note proposes measuring how much analog hardware noise a generative audio model can absorb before people notice, and comparing that with how much noise objective metrics say it can absorb. The question goes beyond scoring. If listeners forgive some kinds of hardware noise (for example, coarse output quantization) and not others (for example, slow weight drift), then the precision targets we give device engineers are wrong for this workload. That matters because some of those targets are expensive to meet. One example is flattening internal electric fields to stabilize stored states, using dielectric or geometric field shaping borrowed from power semiconductors. Such field shaping costs fabrication complexity and adds parasitic capacitance.
Core Claim
Our claim is that, for generative audio, the hardware noise tolerance implied by objective metrics differs from the tolerance implied by human listening, and that it differs by noise type, not by a uniform offset. Two consequences follow. First, training a model to tolerate hardware noise while optimizing a perceptually grounded loss should let us lower converter resolution or accept more drift at equal perceptual quality. Second, the ranking of which device nonideality is most worth engineering away may reverse between the objective view and the perceptual view.
Prior work supplies the pieces but has not joined them. Hardware-aware training injects realistic phase-change-memory crossbar noise into the forward pass and recovers iso-accuracy on five of eleven workloads. It finds that output-referred noise, such as converter quantization, hurts far more than weight noise. Its audio workloads, however, are speech recognizers scored by word error rate; it never generates audio and never asks listeners. The perceptual side shows that objective metrics break on generative output. ViSQOL v3 documents full-reference failure on resynthesizing codecs. Hung et al. show that distribution metrics track spectral texture rather than rhythmic structure. SongEval and CMI-RewardBench show that learned evaluators trained on human ratings beat generic metrics. None of this perceptual work touches hardware. We treat hardware-aware training as the method we reuse, the perceptual literature as the evidence motivating a different yardstick, and device field shaping as the downstream consumer of the result.
Research Program
The first stage builds a measurement instrument. We take a generative model with a published human evaluation: the StyleGAN2 one-bar drum loop generator of Hung et al., trained on the FreeSound Loop Dataset and Looperman. We map its layers onto simulated crossbars in the IBM Analog Hardware Acceleration Kit used by Rasch et al..
This setup has a useful property. Because a generator maps a fixed random input to a fixed output, the full-precision digital model produces an exact reference for every analog sample. That lets us measure the simulation-to-hardware gap directly, one sample at a time. We can run full-reference metrics (signal-to-distortion ratio, mel-spectrogram error, ViSQOL v3) against the digital twin. Alongside them we run reference-free metrics (Inception Score, Fréchet Audio Distance), learned perceptual evaluators, and human listening tests.
We then sweep noise sources one at a time and in combination. The sources are converter resolution, programming stochasticity, conductance drift over simulated time (the power-law drift model in the kit), read noise, and line-resistance voltage drop. Along each axis we record the noise level at which each metric declares a meaningful degradation.
The second stage tests whether the perceptual budget can be spent. We retrain the generator with hardware-aware training under two losses at identical noise injection. One is the conventional sample-level reconstruction-to-twin loss; the other is a perceptually grounded loss built from multi-scale spectral and pretrained-audio-feature distances. For both, we search for the lowest converter resolution and the highest drift exponent at which listeners cannot tell analog output from digital output.
Learned evaluators are vulnerable to reward hacking, which the perceptual literature flags as a known failure. We therefore never use the evaluators that score the final results as training signals. Final judgments rest on held-out evaluators and human tests.
The third stage translates the result to devices, and this is where field shaping enters. Field-flattening techniques target specific noise axes. In the vertical organic electrochemical transistor, a channel depth-to-length aspect ratio near 2,000 flattens the internal field, stabilizes trapped ions, and yields 1,024 stable conductance states. That paper's validation is single-device measurement plus idealized system simulation. Dielectric RESURF structures such as Variable-K trenches and SIPOS pillars show how adjacent dielectrics can reshape field profiles. The same work reports a cost: added parasitic capacitance and slower switching.
We will not fabricate devices. Instead we express the stage-two results as an exchange rate, stated as how many converter bits one unit of drift-exponent reduction is worth under objective versus perceptual criteria. From that we state which device investments the perceptual budget justifies. We are explicit that the link from field uniformity to specific noise parameters is a hypothesis drawn from the v-OECT result, not a calibrated model. Its TCAD-to-silicon gap is itself an open problem noted in the RESURF literature.
Baselines
The naive reference is direct mapping: the full-precision generator deployed onto simulated crossbars with no retraining. Rasch et al. show this fails for classifiers, with WideResNet-16 error rising from 20% to 25%. Whether it fails audibly for a drum generator is the first thing we measure. The full-precision model is the ceiling. Hung et al. report an Inception Score of 5.24 on FreeSound Loops against 6.30 for real data, and an Inception Score of 6.08 with Fréchet Audio Distance 2.22 on Looperman. Their 140-person listening test found no significant difference from real loops in "Drumness" or "Preference". We reproduce that protocol rather than trust our retrained numbers to match.
The strongest existing method is standard hardware-aware training as released in the kit, with the generator's usual objective plus reconstruction toward the digital twin. It is the strongest competitor because it is the only published technique shown to close the simulation-to-hardware gap across diverse networks. No reported result was found on Lacuna for hardware-aware training on any generative audio model, so we establish this baseline ourselves.
Fairness rests on three controls. All arms share the same noise models, data, number of training steps, and hyperparameter search budget. They differ only in the loss. Converter resolution and drift settings are swept identically.
The key ablation isolates the perceptual loss by holding everything else fixed. A second ablation replaces the perceptual loss with a learned evaluator used as a reward, to expose hacking.
Beating the strongest baseline means one of two outcomes at matched human-rated indistinguishability from the digital twin. The first is at least one fewer converter bit, or a tolerated drift exponent clearly above the baseline's. The second is significantly better human preference at matched hardware settings. Either must be confirmed by a held-out human test, not only by evaluators.
Why not just…?
Why not just use digital INT8 or INT4 quantization and avoid analog noise altogether? Digital quantization has no simulation-to-silicon gap, and for many deployments it is the right answer. We include it as a reference arm at matched effective precision. The reason to study analog hardware is energy: Rasch et al. attribute most inference energy to moving weights, which in-memory computing avoids. Our question is narrower, though. It is whether perceptual criteria change the precision analog hardware must deliver, and digital quantization does not answer that. If the digital arm matches analog perceptual quality at lower converter-equivalent cost, we will report that plainly.
Why not just run standard hardware-aware training and score it with a better objective metric like ViSQOL v3? This is the real threat. If ViSQOL v3, or mel distance measured against the digital twin, tracks human judgments of analog degradation well, the perceptual machinery adds nothing. The digital-twin reference removes the alignment problem that ViSQOL v3 identifies for resynthesis, so the alternative is stronger here than in the codec setting. Stage one settles it directly by computing per-noise-axis correlation between each objective metric and listener ratings. A high, axis-uniform correlation would refute our core claim.
Why not just optimize a learned perceptual evaluator such as UTMOS or CMI-RM directly? The perceptual literature warns that generators exploit learned reward heads. CMI-RewardBench also reports high correlation between its evaluation dimensions, which makes targeted control hard. We include direct reward optimization as an ablation and judge it only with held-out evaluators and listeners. If it wins there too, we adopt it.
Why not just sweep noise parameters and skip the device question? The sweep is most of the work. The device stage adds no new experiment, only an interpretation. The interpretation matters because field-shaping engineering trades stability against capacitance and fabrication cost, as Cao et al. show. A precision budget that misstates which noise matters sends that engineering effort to the wrong place.
Proposed Experiments and Decisive Checks
The decisive first experiment is the stage-one tolerance map on FreeSound Loops and Looperman. For each noise axis we report the threshold at which signal-to-distortion ratio, mel error, ViSQOL v3, Inception Score, and Fréchet Audio Distance each flag degradation, alongside the threshold at which listeners do. Listeners take an ABX test (identify which of two clips matches a reference) against the digital twin, plus the Drumness and Preference ratings from Hung et al.. Thresholds come with bootstrap confidence intervals. The core claim survives only if the perceptual threshold sits significantly beyond the objective threshold on some axes and not others. We also test, as a falsifiable prediction, whether output quantization noise is more forgivable perceptually than its dominance in classification degradation suggests. The rationale is that broadband error may be masked in dense percussion, while drift-induced timbral and timing shifts are not.
The second experiment compares the retraining arms in stage two and reports the lowest converter resolution and highest drift exponent that reach ABX indistinguishability for each. Learned evaluators trained in the SongEval style, whose reported human correlation is 0.912, serve as cheap screening during sweeps. Every headline claim is rechecked with fresh listeners.
The third experiment reports the drift-versus-bits exchange rate under objective and perceptual criteria and states whether it reverses the priority between stabilizing stored states and improving readout. If the two rates agree, the device-engineering implication collapses, and we say so.
References
- Hardware-aware training for large-scale and diverse deep learning inference workloads using in-memory computing-based accelerators
- ViSQOL v3: An Open Source Production Ready Objective Speech and Audio Metric
- A Benchmarking Initiative for Audio-domain Music Generation using the FreeSound Loop Dataset
- SongEval: A Benchmark Dataset for Song Aesthetics Evaluation
- CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction
- An organic electrochemical transistor for multi-modal sensing, memory and processing
- A TCAD Study on High-Voltage Superjunction LDMOS with Variable-K Dielectric Trench
- Machine Learning-Based Figure of Merit Model of SIPOS Modulated Drift Region for U-MOSFET