The best machine translation systems now combine outputs from several engines. Some are open models anyone can download. Others are commercial models reachable only through paid interfaces, known as APIs, such as GPT-4o and Claude-3.5. Combining their candidate translations into one output reliably beats any single engine. The catch is that the published number cannot be reproduced by anyone without the same paid access, and it may not be reproducible at all once the provider updates or retires the model. This is the old licensing wall of computational science: you may be allowed to run a proprietary component, but you are not allowed to redistribute it. Earlier it applied to MATLAB and closed decoders. It now reaches ensemble research through commercial APIs.
We propose to treat this as a measurable quantity, the licensing gap. It is the quality difference between a hybrid open-plus-closed ensemble and the best artifact a third party could legally download and run on new inputs. We then test whether the gap can be closed by distilling the hybrid ensemble's combined output into a single open student model. Distillation means training one model to imitate the outputs of another system. The core question is whether the commercial members contribute anything that survives distillation, beyond what an equally expensive open-only ensemble would provide. If the answer is no, hybrid ensembles are mainly a way to buy diversity, and the field could stop reporting results that cannot be reproduced. If the answer is yes, we get a concrete recipe and an honest number for how much capability crosses the wall.
Core Claim
We claim three things. First, most of the gain a commercial model adds to a hypothesis-combination ensemble comes from diversity rather than from something only that model can supply. Second, much of that gain can be carried into a redistributable open model by distilling the ensemble's fused outputs, not the commercial model's raw outputs. Third, the part that cannot be carried over is measurable, and it varies with domain: it should be larger for long-form literary text than for news. What is new here is not ensembling and not distillation, which both exist. It is using the licensing boundary as the experimental variable. We decompose the hybrid gain into a part attributable to diversity, which open models can replace, and a part attributable to the closed members themselves. We also report a reproducibility-over-time measurement for the hybrid numbers that current papers present as fixed.
Relation to Prior Work
Two papers show that hybrid ensembles are where the quality is, and both leave the licensing question open. The Huawei WMT24 system (Choose the Final Translation from NMT and LLM Hypotheses Using MBR Decoding) pools candidates from a deep Transformer translation model and a fine-tuned Llama2-13B. It picks among them with minimum Bayes risk (MBR) decoding, which chooses the candidate that agrees most with all the others under a quality metric. Every component is open or in-house, so that system never crosses the wall. The SJTULoveFiction literary system (Final Submission of SJTULoveFiction to Literary Task) does cross it. It merges Qwen2-72B, Claude-3.5 and GPT-4o through a GPT-4o editing agent, and the result cannot be packaged or rerun without paid access. We take their setting as our hardest case and ask what survives when the closed parts are removed.
The combination methods come from a separate line of work that assumes the candidate translations already exist on disk. Jane (Jane: Open Source Machine Translation System Combination) aligns candidates into a confusion network, a lattice of word alternatives, and votes along it. QE-fusion (Don't Rank, Combine!) recombines divergent spans under a reference-free quality score. It produces a sentence absent from the candidate pool more than half the time. SmartGen/SmartGen++ (Faster Machine Translation Ensembling with Reinforcement Learning and Competitive Correction) learns which subset of models to run for each input and repairs weak candidates. We reuse QE-fusion and MBR as fusion operators. SmartGen's "weakest link" finding, where one mediocre candidate cost over 17 BLEU, constrains our design: fused teacher outputs must be filtered before distillation.
The transfer mechanism comes from Ensemble Distillation for Neural Machine Translation. That paper distilled a six-seed homogeneous ensemble into one student that matched it, a 2.0 BLEU gain over baseline on WMT16 de→en. It also filtered noisy teacher outputs with a translation edit rate threshold. All of its teachers were open and architecturally identical. We extend the method to heterogeneous teachers that sit on both sides of a licensing boundary.
The licensing literature motivates the framing and supplies the reproducibility design. Mood at work: Ramses versus Pharaoh removed a closed decoder by cloning it. The clone matched quality but ran about twice as slow, and cloning a frontier LLM is not feasible. OSIRRC 2019 (Overview of the 2019 Open-Source IR Replicability Challenge) respected data licenses by mounting licensed data into containers at runtime instead of bundling it. We copy this pattern so that closed engines become optional runtime mounts. The speech-VM study (Sharing Speech Synthesis Software for Research and Education Within Low-Tech and Low-Resource Communities) documents the equity cost of this wall: MATLAB-dependent VMs could not be shared, so low-resource users were excluded. Finally, Blameless Users in a Clean Room shows that statistical "access-freeness" arguments do not establish that a model is untainted by its sources. This constrains what we may claim. We say the distilled student is executable and redistributable as software. We do not say it is legally clean of its teachers. Provider terms of service on training with outputs, which are not covered in our materials, must be checked for each provider, and we will use only providers whose terms permit this.
Research Program
The first piece is a small, hook-based harness in the style of OSIRRC, with steps for generate, fuse, distill and evaluate. Each engine is a container. Closed engines are declared as optional external mounts that need the user's own credentials. Every run logs which members were present, the API model identifier and date, and cost. The same fusion code therefore runs in "full hybrid," "open only," and "student only" modes.
The second piece is the measurement. For a quality metric $Q$, we define the licensing gap as $G = Q(\text{hybrid ensemble}) - Q(\text{best redistributable artifact})$. We split it using an open-only ensemble with a matched budget. The diversity share is how much of $G$ disappears when the closed members are replaced by additional open models and samples, with the number of candidates and total compute held fixed. The closed-specific share is the remainder. The distilled recovery is the fraction of $G$ recovered by a student trained on fused hybrid outputs, minus the fraction recovered by a student trained on fused open-only outputs.
The third piece is consensus distillation. We run the hybrid ensemble with QE-fusion over source-side text from the WMT training corpora and filter the fused outputs by a quality-estimation threshold, following the TER filtering of Freitag et al. We then fine-tune an open student of fixed size on the result. The student's training data and weights are the released artifact.
Finally, we rerun the closed members on the same test sets at two dates several months apart. This measures how much the hybrid result itself drifts. That turns the "you cannot reproduce this" complaint into a number.
Baselines
The naive reference point is the best single open model, run alone with plain beam search, with no ensembling. Any artifact that fails to beat it has recovered nothing.
The strongest existing competitor on quality is the full hybrid ensemble itself. On WMT23 en→zh we build the Huawei recipe (NMT plus LLM candidates under COMET-utility MBR) and add closed-API candidates. The Huawei paper reports COMET 0.7234 for its combined system, against 0.7178 for NMT+MBR and 0.7102 for LLM+MBR. Our reconstruction must land near that figure before the gap is meaningful, since we may not match their exact COMET version or models. On the literary task the reference is the SJTULoveFiction merged pipeline, which reports LQR5 of 24% against 9% for GPT-4o alone. We cannot run their human ratings at scale, so we report automatic proxies and a small human study. For QE-fusion we found no reported absolute quality score on Lacuna, only the "over 50% novel outputs" result. For SmartGen++ the reported 62.77 BLEU on Hindi→English uses a private dataset, so it cannot serve as a comparison. All systems use the same test sets, the same number of candidates per segment, and the same fusion code.
The strongest redistributable competitor is an open-only ensemble given the same compute and API-equivalent budget: more open models, more samples each, the same fusion. The key ablation is the distilled student trained on fused open-only outputs against the same student trained on fused hybrid outputs. The two students have identical architecture, training steps, filtering threshold and source text. This isolates what the closed members add after distillation. Beating it means that the hybrid-teacher student is significantly better than the open-teacher student on a held-out metric not used as the fusion utility. We use chrF and a small blind human preference test, because using COMET for both selection and evaluation would reward the fusion for optimizing the evaluation metric itself.
Why not just…?
Why not just distill GPT-4o directly into an open model? This is the obvious move, and we include it as an ablation: a student trained on single closed-model outputs, matched for data size. SmartGen's weakest-link result and the SJTU jump from 9% to 24% suggest that fused outputs are better teachers than any single engine. But whether that advantage survives distillation is exactly what we test. If the single-teacher student matches the fused-teacher student, we will report that the ensemble step is unnecessary for transfer.
Why not just release the cached closed-model translations and let others rerun Jane or QE-fusion on them? This reproduces the published test number, and we will release such caches where terms allow. It is a real, cheap partial solution. It does not produce a system that translates new inputs, and it hides drift. Our two-date rerun measures how stale such caches become, and the student is the only artifact of the three that runs on unseen text.
Why not just use more or bigger open models and drop closed ones entirely? This is the most serious threat to the proposal, and the matched-budget open-only ensemble is built to test it. If the diversity share is close to all of $G$, the conclusion is that closed members are replaceable. That is a useful negative result: papers should report open-only ensembles, and the distillation step is optional. We regard that outcome as plausible for news text and less plausible for literary text.
Why not clone the closed engine, as RAMSES did? RAMSES worked for a phrase-based decoder with a known algorithm and still ran about twice as slow. Frontier LLMs have unknown weights and training data, so there is nothing to clone. Distillation copies the behavior on a task distribution, not the implementation.
Proposed Experiments
We evaluate on WMT23 general en→zh (to anchor to the Huawei numbers), WMT22 en→de, zh→en and de→fr, WMT21 is→en as the lower-resource case (the QE-fusion test suite), and the WMT24 GuoFeng literary task for long-form text. Candidate pools include two or three open models (an NMT system, an open LLM such as Qwen or Llama) and one or two closed APIs, 25 candidates per segment. The 25 figure follows QE-fusion's report that near-oracle performance arrives by that pool size. We report COMET and COMETKIWI, chrF, BLEU (to expose the BLEU–COMET discordance Huawei observed), cost in dollars and GPU-hours, and, on the literary task, cross-chapter term variance as defined by SJTU.
The decisive checks are three. First, if the diversity share exceeds most of $G$ on news but not on literary text, the domain-dependence claim holds. Second, if the hybrid-teacher student beats the open-teacher student on chrF and human preference, closed contributions do cross the wall through consensus distillation. If it does not, they do not, whatever COMET says. Third, the two-date rerun of the closed members shows whether hybrid test scores move by more than the gaps between competing methods. If they do, currently published hybrid comparisons are not stable enough to rank systems, and the redistributable student becomes the more trustworthy reference.
References
- Choose the Final Translation from NMT and LLM Hypotheses Using MBR Decoding: HW-TSC's Submission to the WMT24 General MT Shared Task
- Final Submission of SJTULoveFiction to Literary Task
- Jane: Open Source Machine Translation System Combination
- Don't Rank, Combine! Combining Machine Translation Hypotheses Using Quality Estimation
- Faster Machine Translation Ensembling with Reinforcement Learning and Competitive Correction
- Ensemble Distillation for Neural Machine Translation
- Mood at work: Ramses versus Pharaoh
- Overview of the 2019 Open-Source IR Replicability Challenge (OSIRRC 2019)
- Sharing Speech Synthesis Software for Research and Education Within Low-Tech and Low-Resource Communities
- Blameless Users in a Clean Room: Defining Copyright Protection for Generative Models