Multichannel Generative Language Model: Learning All Possible Factorizations Within and Across Channels

Harris Chan, Jamie Kiros, William Chan

Introduction

Current Natural Language Processing (NLP) models are largely specialized. Translation models are typically trained to predict a target sequence given a source ($p(\text{target}|\text{source})$), while language models focus on next-token prediction ($p(\text{next\_token}|\text{context})$). These architectures are often rigid, relying on fixed autoregressive orders that limit their flexibility in handling multi-language tasks or partial observations.

The Multichannel Generative Language Model (MGLM) addresses these limitations by treating different languages as "channels" of a shared underlying meaning. By learning to generate these channels in any order, a single MGLM can perform unconditional generation, standard translation, and complex multi-channel "in-filling" tasks.

MGLM completes partial sentences across multiple languages simultaneously.

Methods: A Unified Generative Framework

The core of MGLM is its ability to learn the joint distribution of multiple channels by marginalizing over every possible token factorization.

Joint Probability Factorization

Instead of a fixed left-to-right order, MGLM optimizes a lower bound of the joint log-likelihood $p(x_1, \dots, x_k)$ across all channels. It considers all possible permutations ($z$) of tokens, allowing the model to learn dependencies regardless of sequence position or language.

Insertion-Based Architecture

MGLM utilizes an insertion-based approach, building on the Insertion Transformer and KERMIT. Unlike standard Transformers that use causal masking, MGLM predicts a (content, location) pair. This allows it to "fill in the blanks" anywhere in the sequence, facilitating non-monotonic generation.

Unified Decoder and Inference

The model employs a standard Transformer decoder where all channels are concatenated into a single sequence, separated by [SEP] tokens. Each channel is identified by a unique learnable embedding. During inference, the model can fix specific channels for translation, leave them empty for generation, or provide partial snippets for multi-channel completion.

MGLM training and inference procedures, highlighting token/positional embeddings and parallel decoding iterations.

Key Insights

  • The "Channel" Philosophy: By treating languages as different views of the same concept, the model develops a deeper semantic representation independent of specific syntax.
  • Parallelism via Insertion: The ability to insert tokens into any slot allows MGLM to generate multiple languages in parallel, breaking the linear bottleneck of traditional autoregressive translation.
  • Quality-Diversity Trade-off: MGLM achieves a superior balance between translation quality (BLEU) and sample diversity (Self-BLEU) compared to specialized bilingual models, particularly when "fantasizing" missing information.

Key Results

The researchers evaluated MGLM using the Multi30K dataset (English, French, German, and Czech).

  • Competitive Translation: MGLM performs on par with state-of-the-art bilingual models. For English $\rightarrow$ German, the Multitarget MGLM achieved a 36.83 BLEU score, outperforming the specialized baseline of 36.14.
  • Significant Speedup: When generating three target languages simultaneously, MGLM achieved nearly a 3x wall-clock speedup over serial generation due to its parallel decoding capabilities.
  • Resource Efficiency: A single MGLM can replace multiple specialized models, saving significant memory and training resources.
  • Self-Consistency: In unconditional sampling, the model maintains high semantic consistency across generated languages, as evidenced by strong "pseudo-target BLEU" scores.

Conclusion and Future Work

MGLM demonstrates that a unified generative framework can match or exceed the performance of specialized models while offering far greater flexibility. While currently limited to four text channels, the "multichannel" framework is theoretically applicable to other modalities like images or audio. Future research will focus on scaling the model to dozens of languages and exploring its utility in multi-modal tasks.

Summary updated

Abstract

A channel corresponds to a viewpoint or transformation of an underlying meaning. A pair of parallel sentences in English and French express the same underlying meaning, but through two separate channels corresponding to their languages. In this work, we present the Multichannel Generative Language Model (MGLM). MGLM is a generative joint distribution model over channels. MGLM marginalizes over all possible factorizations within and across all channels. MGLM endows flexible inference, including unconditional generation, conditional generation (where 1 channel is observed and other channels are generated), and partially observed generation (where incomplete observations are spread across all the channels). We experiment with the Multi30K dataset containing English, French, Czech, and German. We demonstrate experiments with unconditional, conditional, and partially conditional generation. We provide qualitative samples sampled unconditionally from the generative joint distribution. We also quantitatively analyze the quality-diversity trade-offs and find MGLM outperforms traditional bilingual discriminative models.
BibTeX
@inproceedings{chan2020multichannel,
  author = {Harris Chan and Jamie Kiros and William Chan},
  title = {{Multichannel Generative Language Model: Learning All Possible Factorizations Within and Across Channels}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=iA2epNqKEsJ},
}
Figure