Where Frame Compression Breaks: Direct Speech Translation on Analog In-Memory Hardware

Speech translation models that go straight from audio to target-language text, without first writing down a transcript, are valued for one thing in particular. They keep acoustic cues such as pitch, intonation and voice quality that a transcript throws away, and they use those cues to resolve ambiguities. Running these models on battery-powered devices is expensive, because ordinary chips spend most of their energy moving numbers between memory and the arithmetic units rather than on the arithmetic. Analog in-memory computing addresses this cost. It stores a network's weights as electrical conductances in a grid of memory cells, and the grid computes a matrix-vector product in place using Ohm's and Kirchhoff's laws. The price is noisy, drifting, low-precision arithmetic.

This note argues that the right way to bring the two together runs through one component: the step that shortens the audio sequence inside the encoder. On analog hardware the weights never move, so the cost of an utterance scales with the number of audio frames that must be converted, stored and attended over, not with model size. Frame compression therefore becomes the main energy lever. The decision about which frames to merge is itself computed from noisy analog activations, however, and a wrong merge can erase exactly the brief pitch events that justify direct translation. We propose to measure this failure and to fix it with a compression gate that is trained to tolerate noise and read out at higher precision. We then test whether the energy savings survive without losing the prosodic advantage over a transcript-based pipeline.

Core Claim

Our claim has two halves. The first half is about cost. In weight-stationary analog execution, the cost of each layer is roughly the number of input vectors times the per-vector cost of converting analog column currents back to digital values, plus the size and lifetime of the attention memory holding the encoder output. The analog-to-digital converters (ADCs) that do this conversion dominate tile power and area. All three terms scale with sequence length. Shrinking the sequence by about 4×, as intermediate CTC compression does, should therefore cut a larger fraction of total energy on analog hardware than the roughly 35% speed and memory gain reported on GPUs (Efficient yet Competitive Speech Translation: FBK@IWSLT2022). CTC (connectionist temporal classification) is a loss that labels each frame with a phonetic unit or "blank".

The second half is about failure. The merge decision comes from an intermediate CTC classifier. If that classifier runs on analog layers, read noise and drift flip its frame labels. Flipped labels cause two kinds of error: merges that average away short pitch or voicing events, and splits that inflate length and give back the energy savings. We predict these errors concentrate on prosodically marked and speaker-dependent segments. If so, the cues that make direct translation worth doing are the first casualties of naive analog deployment. We further claim that a gate trained with injected device noise, with a small CTC classifier head kept at higher precision, recovers both translation quality and prosody-sensitive accuracy at nearly the full energy saving.

Relation to Prior Work

Two results motivate the proposal by showing that analog hardware can run large speech and Transformer workloads. An analog-AI chip for energy-efficient speech recognition and transcription ran a recurrent speech recognizer on 14-nm phase-change memory, reaching 9.258% word error rate against a 7.452% software baseline. It handled recognition only, with no attention and no translation. Toward Software-Equivalent Accuracy on Transformer-Based Deep Neural Networks With Analog Memory Devices showed that BERT stays within 1.29% of software on GLUE after simulated month-long drift. It did so by training with injected device noise, a step called hardware-aware training, and by keeping attention digital. We reuse both of these techniques as the method. Their setting is text with fixed lengths, which means they never face a length-changing operation inside the network.

Two further papers supply the hardware partitioning we adopt. X-Former: In-Memory Acceleration of Transformers places static weights in resistive crossbars and dynamic attention in SRAM. Analog in-memory computing attention mechanism for fast and energy-efficient large language models stores keys and values in charge-based gain cells at about 6.1 nJ per token per head, with a conservative 5 ms retention. That retention limit makes the length and lifetime of the stored acoustic memory a real constraint for translation.

The compression method comes from CTC-based Compression for Direct Speech Translation. It merges runs of frames sharing a CTC label at an intermediate encoder layer and improves MuST-C BLEU over fixed downsampling. That work runs entirely on digital hardware and never asks what happens when the labels are noisy.

The case for direct translation that we aim to preserve rests on three studies. Prosody in Cascade and Direct Speech-to-Text Translation: a case study on Korean Wh-Phrases finds that direct models beat cascades by up to 12.9% on pitch-disambiguated utterances. Gender in Danger? Evaluating Speech Translation Technology on the MuST-SHE Corpus finds that direct models exploit vocal cues for gender agreement. Is "moby dick" a Whale or a Bird? Named Entities and Terminology in Speech Translation shows that acoustic detail helps with named entities. No work in the file tests whether these advantages survive analog noise.

One correction to a common framing is in order. Variable-length compression does not force crossbar rows to be re-routed, because weight crossbars process one frame vector at a time regardless of sequence length. The real tension lies in the attention memory and the block pipelining. This is why we treat the compression decision's quality, rather than tensor shapes, as the central problem.

Research Program

We will build a Conformer-style direct translation model with an intermediate CTC collapse block, following the CTC compression paper. We will simulate it in A Flexible and Fast PyTorch Toolkit for Simulating Training and Inference on Analog Crossbar Arrays, the IBM analog toolkit, which provides phase-change memory noise and drift models calibrated on physical chips. All projection and feed-forward weights will be analog. Cross-attention keys and values will be held in a simulated gain-cell memory, with leakage modeled as a function of decoding time.

The first study measures how noisy merging goes wrong. We compare frame labels from the analog CTC head with those from the clean model at each drift horizon. We then report the label flip rate, the change in compressed length, and whether flips cluster on frames carrying large pitch movements, which we identify by pitch tracking.

The second study is the intervention, which has two parts. First, we train the gate with injected noise and add a margin term so that merge decisions stay stable under perturbation. Second, we run only the CTC classifier head at higher digital precision. This head is a single small vocabulary-by-width projection, so its cost is minor.

The third study is an energy model. It counts ADC conversions, digital-to-analog input conversions, attention-memory writes and reads, and memory lifetime per utterance. Per-conversion ADC energy is not reported anywhere in the file, so we sweep it over a wide range and report conclusions that hold across that range. Where possible we anchor the model to the reported 12.4 TOPS/W chip figure and the 6.1 nJ attention figure.

Baselines

The naive reference point is a direct translation model with no learned compression, only the usual fixed convolutional downsampling. It is trained with hardware-aware noise injection in the Spoon et al. style and evaluated in the same simulator. This is what one would get by porting the BERT recipe to speech.

The strongest existing method is digital CTC compression. On MuST-C it reaches 21.9 BLEU on English-German and 23.4 BLEU on English-Italian, as reported in CTC-based Compression for Direct Speech Translation. A stronger training recipe reaches 26.7 BLEU on English-German, as reported in Efficient yet Competitive Speech Translation: FBK@IWSLT2022. We will rerun it at full precision on our own architecture, so that the quality ceiling is our measured number rather than a borrowed one. We will also run it under analog simulation with an analog gate and with ordinary rather than noise-aware training. That second variant is the direct competitor our method must beat.

To measure the prosodic advantage itself, we also need a digital cascade: a speech recognizer followed by text translation, trained on the same MuST-C data. All systems share data, architecture size, number of training updates and decoding settings. Because the analog toolkit trains several times slower than digital training (60 versus 15 seconds per epoch for VGG-8 in its paper), every analog-trained variant gets the same number of wall-clock GPU hours, and the same hyperparameter search is applied to each.

Two ablations isolate the key ingredient. One trains the gate with noise but keeps the head analog. The other keeps the head digital but trains the gate without noise.

Beating the strongest competitor means three things at the one-month drift horizon. Our system must stay within 1 BLEU of the full-precision compressed model, beat the analog-gated compressed model on BLEU, and retain at least the bulk of the direct-over-cascade advantage on prosody-sensitive test sets. It must do all this while keeping compressed length within about 10% of the clean model's.

Why not just…?

Why not just use fixed-stride downsampling, which is static and needs no fragile gate? This is the most serious threat. If analog noise wrecks CTC labels badly enough, fixed striding could match noisy CTC compression on BLEU while being simpler. The digital evidence favors CTC compression, but the analog case is untested. Our naive baseline run at matched sequence lengths, including a fixed 4× stride, settles the question directly. If fixed striding matches our noise-aware gate on both BLEU and the prosody sets, we will report that the gate is not worth it.

Why not just apply hardware-aware training to an uncompressed model? Noise-aware training alone may preserve accuracy, but it cannot remove the per-frame conversion and attention-memory costs. Our energy model quantifies how much is lost. It also measures how often the uncompressed acoustic memory outlives the 5 ms gain-cell retention, which forces refresh writes.

Why not just run a quantized digital model on an edge accelerator? A digital chip still pays to fetch weights for each step. Hardware for machine learning: Challenges and opportunities and From Tokens to Watt-hours: Analytical Energy Estimation for LLM Inference on Modern GPUs identify memory traffic as the dominant cost. The analog speech chip reports a 14-fold efficiency gain over digital accelerators. We cannot measure silicon, so this comparison remains model-based. We include an INT8 digital point in the energy sweep and say plainly where the conclusion depends on the assumed ADC cost.

Proposed Experiments

We will train and evaluate on MuST-C English-German, English-Italian and English-French. Each model is tested at drift horizons of one hour, one day and one month, with several random device instances per horizon. Gender agreement accuracy is measured on MuST-SHE; no reported number for MuST-SHE was found on Lacuna, so our digital direct and cascade systems supply the reference. Prosody sensitivity is measured on the Korean wh-phrase set if its authors make it available. If it is not available, we will use MuST-SHE alone plus a pitch-perturbation probe on MuST-C, in which we flatten the pitch contour and measure how much translations change.

The decisive check combines three measurements. We compute the label flip rate restricted to high-pitch-movement frames. We compute the fraction of the direct-over-cascade MuST-SHE gap that each analog system retains. We then plot energy per utterance against both of these across the ADC-cost sweep. The claim fails under either of two outcomes: flips do not concentrate on prosodic frames, or the noise-aware gate retains no more of the gap than fixed striding does.