Abstract
A channel corresponds to a viewpoint or transformation of an underlying meaning. A pair of parallel sentences in English and French express the same underlying meaning, but through two separate channels corresponding to their languages. In this work, we present the Multichannel Generative Language Model (MGLM). MGLM is a generative joint distribution model over channels. MGLM marginalizes over all possible factorizations within and across all channels. MGLM endows flexible inference, including unconditional generation, conditional generation (where 1 channel is observed and other channels are generated), and partially observed generation (where incomplete observations are spread across all the channels). We experiment with the Multi30K dataset containing English, French, Czech, and German. We demonstrate experiments with unconditional, conditional, and partially conditional generation. We provide qualitative samples sampled unconditionally from the generative joint distribution. We also quantitatively analyze the quality-diversity trade-offs and find MGLM outperforms traditional bilingual discriminative models.
Introduction
Standard language models typically generate text in a rigid, left-to-right autoregressive manner. In machine translation, models are usually trained to map one specific source language to one specific target language, modeling the conditional distribution $p(y|x)$. However, this approach fails to capture the true joint distribution of a concept across multiple representations.
An English sentence, a French sentence, and a German sentence are simply three different "channels" expressing the same underlying meaning. The Multichannel Generative Language Model (MGLM) addresses the challenge of building a single, fully generative model that can handle any combination of observed and unobserved data across multiple channels. By learning all possible generation orders within and across these channels, MGLM moves beyond the constraints of specific source-target pairs or strict left-to-right generation.
Methods: Marginalizing Over Factorizations
To achieve its characteristic flexibility, MGLM marginalizes over all possible factorizations (permutations of token generation) across $k$ channels. Instead of a standard causal architecture, MGLM builds on the Insertion Transformer and KERMIT, utilizing a Transformer decoder without causal masking.
Training and Inference
During training, sequences from all channels are concatenated together, separated by a [SEP] token. The model is trained to observe a partially completed "canvas" of text and simultaneously predict:
- Content: The specific token from the vocabulary to generate.
- Location: Where to insert that token relative to the currently generated words.
Because it is trained on random insertion orders, MGLM learns to generate text in any order. At inference time, it can decode serially (one token at a time) or in parallel, inserting tokens into all available slots across all languages simultaneously. This allows a single model to perform tasks ranging from standard machine translation to filling in partial blanks across multiple languages.
Experimental Results
The authors evaluated MGLM on the Multi30K dataset, which contains 29,000 parallel training sentences in English, French, Czech, and German.
Key Insights
- Efficiency through Flexibility: A single Multi-target MGLM can replace dozens of specialized directional translation models (e.g., EN$→$FR, EN$→$DE).
- Parallel Decoding Speedups: MGLM can translate English into French, German, and Czech simultaneously. When decoding three target languages at once, the model achieves nearly a 3x wall-clock speedup compared to serial decoding. The number of decoding iterations scales logarithmically $\mathcal{O}(\log n)$ rather than linearly.
- Quality-Diversity Trade-off: Multi-target MGLMs achieve a superior trade-off between translation quality (BLEU) and generation diversity (Self-BLEU) compared to specialized bilingual models.
- Competitive Performance: The Multi-target MGLM achieves BLEU scores comparable to dedicated bilingual models (e.g., 36.83 EN$→$DE vs. 36.14 for the baseline on Test2016).
Limitations and Future Work
While highly flexible, fully generative joint models currently lag slightly behind purely discriminative models in strict performance metrics for specific tasks. The model's tendency to "fantasize" details when conditioned on partial sequences makes it creative but occasionally less faithful as a strict translator.
Future work involves scaling MGLM to hundreds of channels and expanding beyond text to multimodal channels, such as images, paraphrases, and question-answer pairs, leveraging the modality-agnostic nature of the "channel" concept.
Summary updated