Emerging non-volatile memories such as resistive RAM and phase-change memory can run neural networks inside the memory array. Each weight is stored as a cell's conductance, and matrix products happen where the data lives. This makes them attractive for always-on speech devices. The catch is that every weight update physically wears a cell, and cells die after roughly $10^6$ to $10^9$ writes. Learning on the chip, which is needed when a voice device must adapt to a new room, microphone, or speaker without sending audio off the device, therefore spends a finite budget with every gradient step.
Speech models that fuse complementary features push in the opposite direction. Combining magnitude features with phase, or handcrafted features with learned ones, repeatedly beats single features under replay attacks and channel shift. Doing so doubles the number of arrays that training must write. We propose a simple co-design that exploits a structural fact. When streams are fused by element-wise addition, which Attention Is Not Always the Answer found to beat cross-attention, the network's output splits exactly into per-stream contributions. That split lets the chip tell, at no extra cost, which stream caused a given error, and then write only that stream's array. We expect this to cut encoder wear several-fold compared with error-gating the whole network, at equal accuracy after adaptation.
Core Claim
Consider a two-stream model with a magnitude stream (for example cepstral coefficients) and a phase stream (for example auditory-filter relative phase). Each stream has its own encoder crossbar. Their outputs are summed into a logit $z = f_m(x_m) + f_p(x_p) + b$. Here $f_m$ and $f_p$ are the stream encoders mapped onto separate memory arrays, $x_m, x_p$ are the two feature views, and $b$ is a bias.
Because the fusion is additive, the fused loss gradient reaching each stream is identical. Ordinary backpropagation therefore writes both arrays on every error, even when only one stream is wrong. We instead compute a per-stream responsibility score:
$$r_s = \ell(z) - \ell\big(z - f_s(x_s) + \bar f_s\big),$$
where $\ell$ is the classification loss and $\bar f_s$ is a running mean of stream $s$'s contribution. A positive $r_s$ means stream $s$ is pulling the decision the wrong way on this example.
The update rule works as follows. The digital fusion head is always updated in SRAM. An encoder array is written only when the fused loss exceeds a threshold $\theta$, and then only for the stream with the largest $r_s$ above a margin $\tau$. The score costs no extra array reads, since $f_s(x_s)$ is already computed in the forward pass and the counterfactual is a subtraction. This carries over the "spend cheap operations to avoid writes" principle from Rethinking Database Algorithms for Phase Change Memory, which traded extra CPU reads for 7.7x to 436x less PCM wear in B+-trees, from data structures to learning.
The claim is falsifiable. At a matched accuracy target after adaptation, stream-selective gating should need substantially fewer total encoder write pulses, and a lower maximum per-cell pulse count, than whole-network error gating. The saving should be largest exactly where complementarity is largest, meaning shifts that corrupt one view but not the other.
Relation to Prior Work
Two papers supply the mechanism we build on. On-Chip Error-triggered Learning of Multi-layer Memristive Spiking Neural Networks gates weight writes with a per-layer error threshold and reports up to 88x fewer updates on DVS Gestures. It treats the network as a single input stream and has no notion of which input view caused an error. We keep its threshold gate and add the cross-stream credit assignment it lacks.
Fast and robust analog in-memory deep neural network training provides the training algorithms (Chopped-TTv2 and AGAD) and the simulator we use. It also motivates the work: it shows that the gradient-accumulation arrays need about $10^4$ times more write endurance than the weights, so any write avoided upstream matters most there. It evaluates only single-stream vision and text models.
The fusion side motivates the regime. Replay attack detection with auditory filter-based relative phase features shows that phase captures playback-device artifacts that magnitude misses. The DKU Replay Detection System reaches its best result only by fusing several representations, and notes that static fusion cannot reweight streams when one is corrupted. This is exactly the situation in which per-stream responsibility should be informative. Tripathi et al. constrain the design: since cross-attention lost to addition on voice activity detection, we do not propose attention-based credit assignment. We treat additive fusion as the design choice that makes exact attribution possible.
The file contains no paper that measures write wear in multi-stream or fused models. The nearest hardware-plus-speech work, ED-sKWS, gates inference timesteps rather than writes and does not simulate memory arrays.
Research Program
We will first build two-stream additive models for replay detection. One is a small convolutional encoder on constant-Q or linear-frequency cepstra. The other is a matching encoder on Gammatone relative phase. The fusion head and classifier stay digital.
We will pretrain these off-chip on a subset of replay conditions, map the encoders into AIHWKit with resistive-RAM and PCM device presets, and adapt on-device to held-out conditions using c-TTv2/AGAD. During adaptation we will log write pulses per cell on both the weight and the gradient-accumulation arrays. Lifetime is estimated as the device endurance divided by the maximum per-cell rate, because the hottest cell kills the array.
We will then analyze when responsibility scores are reliable. The analysis will cover how often the stream with the largest $r_s$ is the one whose standalone probe has degraded, and whether the scores become noisy when both streams are partly wrong.
A second, lighter study repeats this on voice activity detection. There, a frozen pretrained model's projected features are fused additively with cepstral features, and the projection and cepstral encoders are adapted across meeting, telephone, and media domains.
Baselines
The naive reference is no encoder adaptation at all. The pretrained fused model is deployed with frozen encoders, and only the digital head adapts. This costs zero encoder writes and defines the floor that any write spending must beat in accuracy.
The first real competitor is dense on-chip adaptation: plain c-TTv2/AGAD from Rasch et al. updating every array on every example. This sets the accuracy ceiling and the worst-case wear.
The strongest existing method is whole-network error-triggered gating in the style of Payvand et al. Both encoders are written whenever the fused loss exceeds $\theta$. We will sweep its threshold with the same tuning budget, number of seeds, adaptation data, and device models as our method, and compare the two methods along full accuracy-versus-writes curves rather than at single points.
There is no reported result on Lacuna for any on-chip-adapted model on ASVspoof. The accuracy reference points therefore come from software fusion systems. Oo et al. report 9.48% evaluation EER on ASVspoof 2017 for magnitude-plus-Gammatone-phase fusion, against 21.61% for magnitude alone. Cai et al. report 0.66% evaluation EER on ASVspoof 2019 physical access for a six-model fusion, against 2.62% development EER for a single LFCC network.
Beating the strongest baseline means one of two outcomes at equal adaptation EER (within seed variance): at least twofold fewer total encoder write pulses, or at least twofold longer estimated array lifetime.
Two ablations isolate the key ingredient. The first allocates the same number of stream writes at random across the two streams, which tests whether attribution matters or whether halving writes is enough. The second uses a wear-balancing allocator that always writes the less-worn stream, which tests whether our gains are only wear-leveling in disguise.
Why not just…?
Why not just keep the encoders frozen and adapt only the digital head in SRAM? This is the most serious threat. If new replay devices or rooms can be absorbed by re-weighting fixed features, endurance never matters. Our method contains this option as its limit at $\theta \to \infty$, so the experiment settles the question directly. The comparison against the frozen-encoder baseline across held-out conditions will show whether encoder writes buy accuracy.
The phase-feature evidence from Oo et al. suggests that playback artifacts live in low-level structure a linear head cannot re-create. We will nonetheless report the outcome honestly. If head-only adaptation matches dense adaptation within noise, the proposal's regime shrinks to shifts that are genuinely new, and we will say so.
Why not just use whole-network error-triggered gating, which already gives 88x? That 88x figure from Payvand et al. is measured against dense updates on a single stream. It does not tell us how writes split between complementary arrays. Under additive fusion, whole-network gating is guaranteed to write the innocent stream every time the guilty one errs. The matched-budget random-allocation ablation will show whether choosing the right stream adds savings beyond gating itself.
Why not just put the gradient accumulators in digital memory or MRAM? Rasch et al. report that their analog in-memory schemes run 50x to 175x faster than mixed-precision digital accumulation, so offloading gives up the speed that motivates analog training. Moreover, the weight arrays still wear even if the accumulators move. We will also report results with digital accumulation, to show how much of our saving survives on the weight arrays alone.
Why not just rely on controller wear-leveling or rewriting codes? The locally rewritable codes of Kim et al. roughly halve the number of cells modified per update and need parity hardware. Wear-leveling spreads writes but does not remove them. Our wear-balancing ablation measures how much redistribution alone achieves. We view coding as complementary rather than competing.
Proposed Experiments
The main experiment runs on ASVspoof 2017 and ASVspoof 2019 physical access. Pretraining uses a subset of recording and playback configurations, and adaptation targets held-out ones. The metrics are EER, and min-tDCF where available, plotted against total encoder write pulses and maximum per-cell pulses on both weight and accumulation arrays. We use AIHWKit resistive-RAM and PCM presets with five seeds.
The secondary experiment runs on AMI, Callhome, and VoxConverse, adapting voice activity detection sequentially across domains with DER, missed speech rate, and false alarm rate.
The decisive check is a controlled shift that corrupts only one view. We will add phase-scrambling channel distortion that leaves magnitude intact, and magnitude-band noise that leaves phase largely intact. If the responsibility scores work, nearly all writes should land on the corrupted stream's array, and the saving over whole-network gating should approach the fraction of writes that array would otherwise receive. If writes stay split roughly evenly, or accuracy falls behind whole-network gating, the attribution idea fails and we will report that.
References
- Attention Is Not Always the Answer: Optimizing Voice Activity Detection with Simple Feature Fusion
- Rethinking Database Algorithms for Phase Change Memory
- On-Chip Error-Triggered Learning of Multi-Layer Memristive Spiking Neural Networks
- Fast and robust analog in-memory deep neural network training
- Replay attack detection with auditory filter-based relative phase features
- The DKU Replay Detection System for the ASVspoof 2019 Challenge: On Data Augmentation, Feature Representation, Classification, and Fusion
- ED-sKWS: Early-Decision Spiking Neural Networks for Rapid, and Energy-Efficient Keyword Spotting
- Locally Rewritable Codes for Resistive Memories