A Generator-Agnostic Lattice Interface for Swappable Speech-Translation Cascades

Speech translation systems are often built as a chain of two models: a speech recognizer that writes down what was said, followed by a translator. If the recognizer passes along only its single best guess, the translator inherits every recognition error. A known fix is to pass a lattice instead, which is a compact graph whose paths are the many transcriptions the recognizer considered, with a probability on each edge. Lattice-consuming translators do beat 1-best cascades. The catch is that each one is trained on lattices from one particular recognizer, in that recognizer's own segmentation, density and score scale. That defeats the main reason for building a cascade, which is being able to upgrade one module without retraining the other.

This note proposes a fixed contract for the lattice that crosses the module boundary, together with a test of whether it works. The contract has three requirements. Lattices are rewritten into a canonical subword-level form. That rewriting must preserve the probability the recognizer assigned to every path. Edge probabilities are then recalibrated so they mean the same thing regardless of which recognizer produced them. The claim is testable: a translator trained once on canonical lattices from recognizer A should keep most of its lattice advantage when recognizer A is replaced by B at test time, and it should do so without retraining. Raw-lattice translators, by this hypothesis, lose that advantage.

Core Claim

Current lattice translators sit at two ends of a trade-off, and neither treats the lattice as a module interface. Lattice Transformer for Speech Translation builds the lattice structure (shortest-path distances, masks between paths that exclude each other, recognizer marginal probabilities) directly into custom attention. Its ablations found the marginal probabilities to be the most important ingredient, and it reached 39.2 BLEU on Fisher/Callhome Spanish–English against 38.5 for a 1-best Transformer, at about 40% more inference time. Neural Speech Translation using Lattice Transformations and Graph Networks takes the other route. It converts lattices into line graphs, splits words into subwords, minimizes the graphs with Hopcroft's algorithm, and encodes them with a gated graph neural network, a network that updates each node from its neighbors' states. It reached a 39.1 BLEU ensemble with a 144x training speedup, but it explicitly drops the recognizer's edge probabilities. The two results are compatible once one notices a technical point. Unweighted minimization is not merely missing the weights. It merges states whose outgoing path probabilities differ, so the probabilities cannot be correctly reattached after the fact.

The proposal is to replace unweighted minimization with a weight-preserving pipeline and to add a calibration step. The pipeline has three stages: subword expansion, weight pushing (redistributing edge weights so that each node's outgoing weights become conditional next-token probabilities), and weighted determinization and minimization. Every path keeps exactly the probability the recognizer gave it, so the compression advantage of the graph-network approach and the probability advantage of the Lattice Transformer are kept together. The new claim is about modularity. With calibration, the canonical lattice becomes a generator-independent interface, and we predict that this, more than either ingredient alone, is what makes the downstream translator survive a change of recognizer.

Mechanism

Let $w(e)$ be the recognizer's weight on edge $e$ from node $u$ to node $v$. Let $\alpha(u)$ and $\beta(v)$ be the usual forward and backward sums of path weight into $u$ and out of $v$, and let $Z$ be the total weight of all paths. The marginal probability that a transcription passes through $e$ is

$$\gamma(e) = \frac{\alpha(u)\, w(e)\, \beta(v)}{Z}.$$

Weighted determinization and minimization in the log semiring leave the probability of every path unchanged, so $\gamma$ can be recomputed exactly on the canonical graph. The failure mode is size blow-up during determinization. When that happens, we prune by $\gamma$ to a fixed edge budget per second of audio. That budget is part of the contract, so different recognizers deliver lattices of comparable density.

Different recognizers produce marginals on different scales. A neural attention recognizer's scores are typically far more peaked than a conventional recognizer's. For this reason the contract also includes a per-generator temperature $T$, which rescales scores as $w(e)^{1/T}$ before marginals are computed. $T$ is fit on a small held-out set so that $\gamma(e)$ matches the empirical rate at which edges with that marginal lie on the reference transcript. Only this one scalar is fit per new recognizer; the translator itself is never touched.

Downstream, we reuse the line-graph gated graph network of Neural Speech Translation using Lattice Transformations and Graph Networks for speed. The calibrated $\gamma$ enters in two places: it gates each message passed between nodes, and it scales the decoder's attention over nodes in log space. This keeps the lightweight encoder while putting back the signal that the Lattice Transformer ablations identified as most important. The software side follows the base-class discipline of modular toolkits such as DeepDIVA and FaceX-Zoo. There is one interface for lattice generators, which emit weighted acceptors, one canonicalizer, and one interface for lattice consumers. The software is not the contribution, however. The contribution is the measured invariance.

Research Program

We will build three recognizers that produce lattices in genuinely different ways.

These three differ in segmentation, density and score sharpness, which is exactly the variation a module swap introduces.

Translators are trained on lattices from one generator and tested on lattices from every generator. The primary quantity is the swap gap: the BLEU of a translator trained on A and tested on B, minus the BLEU of a translator trained and tested on B. A second quantity is retained lattice gain: the fraction of the matched-condition lattice-over-1-best improvement that survives a swap. Both are evaluated on Fisher/Callhome Spanish–English, where the prior lattice results live, and on MuST-C English–German. For MuST-C, MuST-C: a Multilingual Speech Translation Corpus reports a cascade baseline of 12.25 BLEU and 32.05 WER. That setting tests whether the result transfers to a domain where we generate all lattices ourselves.

Baselines

The naive reference point is a 1-best cascade: a strong text Transformer translating each recognizer's single best transcript. It is trivially swap-robust, because a string is a string. On Fisher/Callhome, the comparable figure reported by Lattice Transformer for Speech Translation is 38.5 BLEU. We rerun it with our recognizers so every comparison uses the same transcripts.

The strongest existing method is the Lattice Transformer itself, which reports 39.2 BLEU on Fisher/Callhome. We reimplement it on the same lattices, data, and hyperparameter search budget as our model. It is the right competitor because it already uses marginals and represents the quality ceiling for in-distribution lattice translation. The graph-network model of Neural Speech Translation using Lattice Transformations and Graph Networks (39.1 BLEU ensemble) is the second strong baseline and the closest structural relative. We compare against it as single models, since our proposal is also a single model. All models share the same subword vocabulary and are tuned on the same development set.

A confusion-network baseline matters because confusion networks are themselves a canonical, recognizer-independent lattice format. They are what Moses accepted. A confusion network collapses the lattice into a sequence of slots of alternative words. We feed confusion networks built from each generator to the same graph encoder.

Three ablations isolate the key ingredients. The first is canonicalization without calibration. The second is calibration on raw, unminimized lattices. The third reattaches posteriors to Hopcroft-minimized unweighted graphs by averaging over merged edges. This third ablation tests whether exact weight preservation matters, or whether approximate weights are just as good.

Beating the baselines means two things at once. First, matched-condition BLEU must be within noise of the Lattice Transformer. Second, retained lattice gain under swaps must be statistically higher than that of the Lattice Transformer, the graph network, and confusion networks, using paired bootstrap tests. The cross-generator swap gap should shrink to a fraction of what raw-lattice models show.

Why not just…?

Why not just pass 1-best strings and keep the interface simple? This is the modular default, and it is perfectly swap-robust. On Fisher/Callhome it loses roughly 0.7 BLEU to lattice models (38.5 vs 39.2), and the same paper's subword-ambiguity experiment on WMT17 Chinese–English shows 24.26 vs 24.81. Our question is whether a richer interface can be made just as swap-robust. If the retained lattice gain under swaps drops to zero, 1-best wins, and we would report that.

Why not just pass an N-best list? Lists are recognizer-independent and supported by every sequence model. The trouble is that a short list spends its slots on near-duplicates. Exploring Recombination for Efficient Decoding of Neural Machine Translation finds that a lattice at the same beam size represents about 200 times more sequences, and Lattice-based Minimum Error Rate Training shows that list-based training sees an impoverished error surface (42.9 vs 43.9 BLEU on Arabic–English). We include N-best reranking with 10 and 50 hypotheses, both under swaps and at matched decoding compute, as a direct check.

Why not just add the posterior as an edge feature to the existing graph network? This is the most obvious fix, and it is a real threat. Our third ablation is designed to settle it. If approximate posteriors on unweighted-minimized graphs, combined with calibration, match exact weight preservation, then the weighted-automaton machinery is unnecessary and the contribution reduces to calibration. We would report it that way.

Why not just use confusion networks, which are already canonical? Also a real threat, because they deliver recognizer independence cheaply. They lose path structure, though: alternatives that cannot co-occur become freely combinable, and this is the kind of dilution that Flexible and Efficient Hypergraph Interactions observed when unconstrained mixing (its "interchange" scheme) underperformed constrained mixing. The confusion-network baseline under swaps directly measures whether the structure is worth keeping.

Why not just retrain the translator for each new recognizer, or train end-to-end? Retraining on B's lattices is our upper-bound condition. If it is cheap and large, the contract buys little. End-to-end models avoid the boundary entirely but need aligned speech–translation data, which MuST-C notes is scarce. The proposal addresses the regime where recognizers are upgraded more often than translators can be retrained.

Decisive Checks

The central experiment is the 3×3 train/test generator grid on Fisher/Callhome, repeated on MuST-C English–German. The proposal succeeds if three things hold together: canonical calibrated lattices keep most of the matched-condition lattice gain under off-diagonal swaps, raw-lattice translators lose most of it, and confusion networks fall between the two.

Three secondary checks follow. The first is calibration quality: the gap between predicted $\gamma$ and the empirical rate of edges lying on the reference, measured per generator before and after temperature fitting. If swap robustness improves as this gap closes, that points to calibration as the mechanism. The second is lattice size and inference time relative to the Lattice Transformer's reported 40% overhead. The third is the swap from generator B to C, which pits exact state merging against heuristic merging. It shows whether the contract tolerates the approximate equivalence that Transformer decoders force on lattice generation.