Speech translation is often built as a cascade: a speech recognizer writes out a transcript, and a separate translator turns that transcript into another language. The weak point is the handoff. When the recognizer commits to one word sequence, any mistake it makes reaches the translator as if it were certain. A known fix is to pass something richer, such as the recognizer's internal summary vectors. On analog in-memory chips, which do multiplication inside the memory array to avoid shuttling weights back and forth, that fix has a hidden cost. Every value crossing from one model to the next must be converted from analog to digital and moved over a bus, and these converters and transfers are where much of the energy goes. A committed word token costs about 15 bits per step. A floating-point summary vector costs thousands. This proposal argues that the number of bits crossing the seam should be a quantity we design and train, not a side effect of whichever interface we happen to choose.
The case is sharper on analog hardware for a second reason. Rounding to a discrete token is not only a source of linguistic error. It also wipes out the device noise and drift that build up inside the recognizer's analog arrays. A rich continuous interface passes that hardware noise straight into the translator. So the seam carries two competing kinds of error at once. We hypothesize that a learned, few-bit interface, quantized at the converter boundary and trained under simulated device noise, can match the robustness of continuous interfaces at a transfer cost close to that of tokens.
Core Claim
Consider an analog speech translation cascade in which the recognizer and translator sit on separate in-memory tiles or chips. Its operating points lie on a curve of translation quality against interface rate $R$, the number of bits per handoff step that cross the analog-to-digital boundary. Hard tokens sit at the low-rate end, around $\log_2 V$ bits for vocabulary size $V$. Continuous attention-context passing sits at the high-rate end, around $d \times b$ bits for vector width $d$ and precision $b$. Neither end is optimal once analog noise is present. We will train the interface jointly with the translation loss and a rate penalty,
$$ \mathcal{L} = \mathcal{L}_{\text{trans}}(\tilde{z}) + \lambda R(\tilde{z}), $$
where $\tilde{z}$ is the handoff code computed from noise-injected recognizer states, $\mathcal{L}_{\text{trans}}$ is the translator's cross-entropy, and $\lambda$ sets the operating point. We claim three things. First, the resulting curve Pareto-dominates hard tokens, N-best lists with scores, and post-hoc quantized continuous vectors. Second, the quantization step at the seam acts as a "noise reset" that measurably limits how far upstream device drift spreads downstream. Third, the best rate rises as drift worsens, which would mean that hardware state should set the interface width.
Relation to Prior Work
The motivating evidence comes from the speech side. Attention-Passing Models replaces predicted text with attention context vectors, which are weighted sums of the recognizer's audio-encoder states. It reports 36.70 BLEU against 32.45 for a text cascade on Fisher Spanish–English, and a 10% versus 26% relative drop under 10% transcript corruption. That work is purely floating-point and ignores what its interface costs to move. We keep its interface shape and put a price on the bandwidth. Speech Translation and the End-to-End Promise supplies the taxonomy of committed versus marginalizing cascades. It also gives the data-scarcity argument for keeping models modular, which constrains us against collapsing everything into one direct model. Lattice translation (Generalizing Word Lattice Translation) is the classic low-rate soft interface, and we treat it as a serious competitor rather than as history.
The hardware side supplies the methods and the constraints. An analog-AI chip for energy-efficient speech recognition and transcription maps a 45M-weight speech transducer onto phase-change memory, reaching 9.258% WER on Librispeech against a 7.452% software baseline. However, it stages auxiliary operations and inter-module data off-chip, and it stops at the transcript. We start where it stops, at the seam it leaves digital. Toward Software-Equivalent Accuracy on Transformer-Based DNNs With Analog Memory Devices gives us its noise and drift model, its hardware-aware training recipe, and its split of projections to analog and attention to digital. It studies BERT classification, not a two-model generative pipeline. CorrectNet is the closest work on analog error propagation. It bounds layer-to-layer amplification with Lipschitz regularization and digital patches on CNNs, recovering VGG16 on CIFAR-100 from 1.69% to 67.01%. Its propagation is inside a single model and its fix adds conversions. Ours is between models, and our fix removes conversions. MemTorch reports an accuracy cliff below 4–5 converter bits, and this constrains how low per-element precision can go. It is one reason we learn a short discrete code instead of simply shrinking every element of a wide vector.
Research Program
The pipeline is an ESPnet-ST-style recognizer and translator (ESPnet-ST). Each is pretrained on its own data and then coupled through an attention-passing seam. Their linear layers run in a simulated phase-change memory crossbar, with programming noise, read noise, and power-law drift following Spoon et al. Attention and softmax stay digital, as in that work.
At the seam, the recognizer's context vector goes through a small projection and then a learned discrete code. The code is either vector-quantized with a codebook of size $K$, giving $\log_2 K$ bits, or a product of $m$ such codebooks, giving $m\log_2 K$ bits. Because the projection is performed inside the analog array, the converters read out only the code, not the full vector. We sweep $R$ from about 8 to about 512 bits per step. Training injects recognizer-side device noise, so the code learns to stay stable under drift.
Energy is accounted analytically. Interface energy is the number of conversions times the per-bit conversion-plus-transfer cost. We leave that cost as a parameter swept over a plausible range, because the file gives no per-conversion energy figure, and we report results across the range. We separately count crossbar-internal conversions, so that interface savings are not mistaken for whole-system savings.
The analysis component measures propagation directly. We perturb the recognizer's analog weights at controlled drift times. Then we record how far the translator's hidden states and output distribution move, for each interface rate. This tests the noise-reset hypothesis without depending on BLEU alone.
Baselines
The naive reference is the committed text cascade with greedy transcripts: the same recognizer and translator, with the seam carrying only token IDs and the same noisy recognizer weights. On clean Fisher, Sperber et al. report 32.45 BLEU for this setup. We expect it to be cheapest at the seam and most fragile to transcription errors.
The strongest existing method is the full-precision Attention-Passing Model (36.70 BLEU on Fisher, Sperber et al.). We reimplement it in the same codebase with the same pretraining data, the same hardware-aware training, and a matched hyperparameter search budget, so the only difference is the seam. Beating it means matching its BLEU within one point under one month of simulated drift, while crossing the seam at least an order of magnitude fewer bits. It also means beating it outright at long drift times if the noise-reset effect is real.
The low-rate soft competitor is an N-best or pruned-lattice interface with scores, fed to a translator trained to consume it. It gets the same conversion accounting and the same tuning budget.
The key ablation is post-training quantization of the attention-passing vectors to INT8, INT4, and INT2. These are matched in total bits to our learned codes but are not trained under noise. It isolates whether the gain comes from learning the code, or merely from transferring fewer bits.
A second ablation removes noise injection from code training. A final reference is a direct speech-to-text model mapped to the same simulated hardware. ESPnet-ST reports 22.91 BLEU on MuST-C En-De for its Transformer baseline. We report it in the low-data condition where Sperber et al. found direct models collapse.
Why not just…?
Why not just pass N-best transcripts or lattices, which are already soft and cheap? This is the real threat. A 5-best list with scores costs on the order of 80 bits per step, which is squarely inside our rate range, and it reuses the noise-reset property of discrete tokens. Our argument against it is only that text hypotheses still discard acoustic cues that the hypotheses do not encode (End-to-End Promise). That is a hypothesis, not evidence. The rate-matched comparison settles it. If lattices land on our curve, the contribution narrows to the propagation analysis and a rate-aware design rule, and we will say so.
Why not just run the digital cascade in INT8 or INT4? The INT6 attention result of Spoon et al. shows how much digital quantization can save. Even so, the 14-fold gain of Ambrogio et al. is for in-memory execution against digital accelerators. More to the point, this proposal concerns the seam that exists in either substrate. We will report the digital INT8 cascade's interface cost alongside ours, so a reader can see whether the learned code also helps there.
Why not just quantize the attention-passing vectors after training? This is exactly the post-training quantization ablation. MemTorch's cliff below 4–5 bits (MemTorch) suggests that cutting per-element precision is the wrong lever compared with cutting the number of elements. If post-training INT4 matches our codes at equal bits, the learned code is unnecessary.
Why not just put CorrectNet-style compensators at the seam? Its digital patches (CorrectNet) read a full analog activation, which adds conversions at exactly the point we are trying to narrow. We include a compensator variant to measure its quality-per-bit directly.
Proposed Experiments
The primary benchmark is Fisher–Callhome Spanish–English, chosen because the attention-passing and cascade numbers above come from it. We also test the low-data 14k-sentence split, where data efficiency matters most. MuST-C En-De (MuST-C) serves as a second language pair, comparing against the 22.91 BLEU ESPnet-ST reference. Librispeech WER verifies that our simulated recognizer degrades in line with the 9.258% versus 7.452% gap reported on real hardware, which calibrates the noise model before any translation claim.
The first decisive check is a BLEU-versus-rate curve at drift times of zero, one day, and one month. The learned code must sit strictly above both the post-training-quantized vectors and the rate-matched lattices at some rate. Otherwise the core claim fails.
The second check is the propagation measurement. We compare translator-state displacement per unit of recognizer drift across rates. The noise-reset hypothesis predicts that this displacement grows with $R$. If it does not grow, the hypothesis is false and low-rate codes earn only their energy saving.
The third check is whether the best rate shifts upward with drift time. If it does, that justifies an interface width that adapts as devices age.
Interface energy is reported across the swept per-conversion cost. Throughout, no absolute joule figure is claimed beyond what the parameterized model supports.
References
- Attention-Passing Models for Robust and Data-Efficient End-to-End Speech Translation
- Speech Translation and the End-to-End Promise: Taking Stock of Where We Are
- Generalizing Word Lattice Translation
- An analog-AI chip for energy-efficient speech recognition and transcription
- Toward Software-Equivalent Accuracy on Transformer-Based Deep Neural Networks With Analog Memory Devices
- CorrectNet: Robustness Enhancement of Analog In-Memory Computing for Neural Networks by Error Suppression and Compensation
- MemTorch: An Open-source Simulation Framework for Memristive Deep Learning Systems
- ESPnet-ST: All-in-One Speech Translation Toolkit
- MuST-C: a Multilingual Speech Translation Corpus