Which Sounds Break First on Analog In-Memory Hardware?

Most of the energy in running a neural network goes to moving weights between memory and the processor, not to the arithmetic itself. This is the "von Neumann bottleneck." Analog in-memory computing avoids it. Each weight is stored as the electrical conductance of a memory cell in a grid called a crossbar. An input voltage is applied to each row, and the currents that add up along each column compute a matrix-vector product in place. The cost is that every product comes back slightly wrong: cells are programmed imprecisely, readings are noisy, stored values drift over hours and days, and the converters that turn currents back into digital numbers have few bits. Universal audio encoders are networks meant to produce one embedding that works for speech, music and environmental sound. They are an obvious workload for this hardware, because always-on listening sensors cannot afford to fetch hundreds of millions of weights per second.

This note proposes to test a specific, falsifiable claim. Analog error does not hurt all sound domains equally. Tasks that depend on fine frequency detail, such as musical pitch and percussion stroke identity, should collapse well before speech-command or environmental-sound tasks. The damage should also be concentrated in a few identifiable layers. If both hold, a universal audio encoder can run mostly in analog form. Training would use a noise-injection recipe balanced across domains, and a small, measured set of pitch-critical layers would stay in digital logic. This should cost far less than keeping the whole network digital, and it should beat the single-domain recipe that has worked for text Transformers.

Core Claim and What Is New

The closest prior result is the work of Spoon et al.. They mapped the linear layers of BERT-base onto phase-change memory crossbars. They trained with injected programming and read noise, compensated for drift, and kept attention in 6-bit integer digital logic. The result was GLUE accuracy within 1.29% of full precision. That work is on text, evaluates one task family at a time, and has no reason to ask whether different kinds of downstream information fail at different rates.

Our claim is that for audio this question is central. The cross-domain audio literature already shows that representations are lopsided. Speech-trained models such as HuBERT discard pitch to gain phonetic invariance. Wu et al. report 42.9% for HuBERT alone on NSynth Pitch, against 88.5% when it is ensembled with the CREPE pitch tracker. Fine pitch information is therefore carried by a thin, fragile part of the representation. Our hypothesis is that analog noise attacks that part first.

The literature also points the other way, and we will report whichever result we find. Spoon et al. and Nandakumar et al. both describe analog noise acting like dropout, a regularizer that helps on small datasets. Some small HEAR tasks, such as Mridangam Stroke or Beijing Opera Percussion, might therefore improve under mild noise rather than degrade.

Nothing in the research file maps a general-purpose audio encoder onto crossbars. Simulation work stops at vision: MemTorch evaluates MobileNetV2 on CIFAR-10. Hardware demonstrations stop at MNIST or text-like sequences: Can Li et al., Nandakumar et al., and the phase-change memory hyperdimensional computing system of Karunaratne et al., which handles language and muscle-signal gestures but no audio. The novelty here is therefore twofold. The first part is a per-domain, per-layer sensitivity map of a universal audio encoder under realistic analog error, measured over time since programming. The second is a placement rule, driven by that map, for deciding which layers stay digital.

Research Program

We study two encoders that cover the two main audio architectures. The first is PANNs CNN14 (Kong et al.), a convolutional network trained with labels on AudioSet. It reports 0.439 mean average precision there. Its convolution and linear layers convert directly through MemTorch's patching of standard PyTorch layers. The second is a speech-pretrained Transformer of the HuBERT or wav2vec 2.0 family. Following Spoon et al., its projection and feed-forward layers go to analog crossbars and attention stays in digital logic.

MemTorch supplies the non-idealities. These are finite conductance levels, device-to-device variability, stuck cells, conductance drift, and converter resolution between 3 and 8 bits. Each crossbar tile is 128×128 or 256×64. We use the tiling geometries MemTorch already studies because wire parasitics worsen with array size.

The program has three stages. The first stage is diagnosis. We freeze each encoder and simulate it in analog form. We extract embeddings and score them with the unchanged HEAR protocol (Turian et al.), in which a small MLP is trained on frozen embeddings, and with the ARCH linear-probe protocol (La Quatra et al.). We do this at several simulated times after programming, ranging from seconds to about a month. The probe is trained on embeddings at the earliest time and then tested at later times, because that is how a deployed sensor would actually age.

The second stage is localization. We switch analog error on one layer at a time and record the drop in each domain's score. This gives a layer-by-domain sensitivity matrix. We test whether the pitch and music rows are concentrated in a few layers or spread across the network.

The third stage is a remedy, which has two parts. One is hardware-aware training on a balanced mixture of speech, music and environmental clips. Hardware-aware training here means fine-tuning while simulated analog noise is injected into the forward pass. The mixture replaces training on AudioSet alone. The other part keeps the $k$ most music-sensitive layers digital, with $k$ chosen so that the analog-mapped fraction of multiply-accumulates stays at or above a fixed budget, for example 90%.

Energy is estimated by counting the operations each design performs: converter conversions, digital multiply-accumulates, and off-chip weight fetches. These counts are weighted with per-operation costs from the hardware literature the file summarizes, including Sze et al.. This is an estimate, not a silicon measurement, and the note says so wherever energy numbers appear.

Baselines

The naive reference is the full-precision digital encoder scored with the same HEAR and ARCH probes. It sets the ceiling for every task. A second naive point is the same encoder mapped to analog with no hardware-aware training at all. It shows how much damage the untreated mapping does.

The strongest existing method is the recipe of Spoon et al., applied to audio as faithfully as possible. It combines noise injection during training, reference-based drift compensation, attention kept in 6-bit integer digital logic, and training on the encoder's original pretraining distribution, which is AudioSet for CNN14 and speech for HuBERT. On text it reports aggregate GLUE accuracy within 1.29% of full precision. We found no reported result on Lacuna for any audio encoder on analog crossbars, so this recipe's audio numbers will be produced here. It receives the same simulator settings, the same fine-tuning compute and the same hyperparameter search budget as our method. Beating it means two things at an equal analog fraction and equal training compute. First, it means a smaller gap to full precision on the music tasks (NSynth Pitch, Mridangam Stroke, Beijing Opera Percussion) at the one-month drift point. Second, it means no more than one point of loss relative to it on speech and environmental tasks.

A strong digital competitor is integer post-training quantization of the same encoder to 8 and 4 bits. We found no reported HEAR result on Lacuna for quantized audio encoders, so we run this baseline ourselves with the same probes. It separates rounding loss, which both digital and analog designs suffer, from the noise and drift that only analog designs suffer.

The key ablation isolates the placement rule. It keeps the same number of digital layers but chooses them in one of two other ways: at random, or by a domain-blind criterion that ranks layers by their average sensitivity across all tasks. If the music-sensitivity rule does not beat both, the claim that damage is domain-concentrated fails. A second ablation trains with balanced data but keeps everything analog. It shows whether the data mixture alone is enough.

Why not just…?

Why not just quantize to 4-bit integers and run on a digital edge processor? This is the most serious alternative. If 4-bit digital encoders lose as little as the analog ones, the accuracy part of our argument rests only on energy. Our simulation can estimate energy but cannot settle it. Sze et al. report 2.56× energy savings from reduced precision in digital accelerators, yet data movement still dominates. Karunaratne et al. report more than a 6× end-to-end energy reduction for a full in-memory system on a different workload. Neither figure transfers directly to audio encoders. The 4-bit baseline and the operation counts will show where each design sits on an accuracy-versus-estimated-energy curve. If the digital 4-bit design dominates that curve, we will report that. The per-domain sensitivity map remains useful either way, since quantization may also hit pitch first.

Why not just use the Spoon et al. recipe unchanged? It may be enough. If it already preserves music tasks under drift, then audio does not need special treatment, and that is a publishable negative finding. The head-to-head comparison in the Baselines section decides it. Our reason to expect a gap is that its training data is single-domain. Wang et al. report that scaling within one domain does not transfer to others, and noise-robust training on AudioSet gives no gradient signal to protect pitch.

Why not just run small domain specialists that fit in on-chip memory? A keyword spotter and a sound-event classifier avoid off-chip fetches, but they do not cover unexpected sounds. La Quatra et al. show speech-only pretraining losing nearly 12 points on UrbanSound8K and 14% on MS-DB. Wu et al. show the 42.9% pitch blind spot of a speech model. Specialists answer a narrower question than the one a general listening sensor faces. We include a specialist pair's estimated memory footprint in the energy accounting so the trade-off is visible.

Why not just keep the whole network digital wherever it is sensitive? Keeping everything sensitive digital gives up the bottleneck savings the hardware exists for. The budgeted placement rule is exactly the test of how few digital layers are needed.

Decisive Checks

Three results would confirm the proposal. The first is that, in the diagnosis stage, the music tasks lose clearly more of their full-precision score than Speech Commands and ESC-50 do at matched noise and drift. The second is that the sensitivity matrix shows the music loss concentrated in a minority of layers. The third is that the domain-aware placement plus balanced training beats both the Spoon-style baseline and the random and domain-blind placement ablations on music tasks at one month, without losing on speech and environmental tasks.

Two results would refute it. One is uniform degradation across domains, which would mean the domain-specific framing adds nothing. The other is a regularization benefit on the small music tasks, which would reverse the predicted direction. A small control test also comes from the data itself. Gunshot Triangulation has only 88 clips, so the probe's own variance must be estimated with repeated seeds before any per-task difference is called real.