Language Models are General-Purpose Interfaces

Yaru Hao, Haoyu Song, Li Dong, Shaohan Huang, Zewen Chi, Wenhui Wang, Shuming Ma, Furu Wei

Introduction

In the current AI landscape, a curious paradox has emerged: while the Transformer architecture has become the standard for state-of-the-art models, these systems remain largely specialized. We see distinct models for computer vision, natural language translation, and creative text generation. Microsoft Research's MetaLM (Meta Language Model) framework aims to break these silos by treating a large-scale language model as a universal "operating system" capable of docking with specialized sensors to handle any modality.

MetaLM as a General-Purpose Interface for diverse tasks

The MetaLM Framework: System 1 and System 2

MetaLM is designed as a "best-of-both-worlds" architecture. It draws an analogy to human cognition:

  • System 1 (Perception): This layer consists of pretrained, bidirectional encoders (e.g., BERT for text, BEiT for images). These act as the model's "eyes" and "ears," processing raw sensory data into rich representations.
  • System 2 (Reasoning/Interface): A central causal language model serves as the universal task layer, performing reasoning and generating outputs.

This modularity is a key strength. Because specialized encoders "dock" with the central language model via a connector layer, different modalities can be integrated without retraining the entire core "brain."

The MetaLM architecture connecting specialized encoders to a central semi-causal language model

The Semi-Causal Objective

The technical innovation driving MetaLM is its semi-causal training objective. Traditional causal language models (like GPT-3) predict tokens from left to right, which is excellent for generation but often lacks the deep "understanding" required for complex inference tasks.

MetaLM's semi-causal approach allows the model to autoregressively generate text while incorporating bidirectional representations of specific spans—such as an image or a paragraph—directly into the sequence. This enables the model to "read" the full context of an input while "writing" the response one token at a time, bridging the gap between Natural Language Understanding (NLU) and generation.

Experimental Results

The researchers evaluated MetaLM (1.3B parameters) across a massive suite of 34 NLP datasets and 9 vision-language datasets. Key findings include:

  • Superior NLU Performance: In multitask fine-tuning, MetaLM significantly outperformed standard GPT baselines, particularly in Natural Language Inference (79.1 vs 65.0) and Coreference Resolution (84.3 vs 67.1). It achieved an impressive 91.1% accuracy on the MNLI task.
  • Vision-Language Capabilities: On zero-shot Visual Question Answering (VQA), MetaLM surpassed larger models like the 7B-parameter Frozen model.
  • Generative Excellence: In zero-shot image captioning on the COCO dataset, it achieved a CIDEr score of 82.2, nearly doubling the performance of several specialized competitors.

Conclusion and Future Directions

MetaLM demonstrates that language models can serve as effective general-purpose interfaces for a wide array of AI tasks. By combining the strengths of bidirectional encoding with the flexibility of causal generation, it offers a path toward more unified and modular AI systems. Future work aims to scale the model further and integrate additional modalities such as audio and video, as well as tackling more complex spatial tasks like object detection.

Summary updated

Abstract

Foundation models have received much attention due to their effectiveness across a broad range of downstream applications. Though there is a big convergence in terms of architecture, most pretrained models are typically still developed for specific tasks or modalities. In this work, we propose to use language models as a general-purpose interface to various foundation models. A collection of pretrained encoders perceive diverse modalities (such as vision, and language), and they dock with a language model that plays the role of a universal task layer. We propose a semi-causal language modeling objective to jointly pretrain the interface and the modular encoders. We subsume the advantages and capabilities from both causal and non-causal modeling, thereby combining the best of two worlds. Specifically, the proposed method not only inherits the capabilities of in-context learning and open-ended generation from causal language modeling, but also is conducive to finetuning because of the bidirectional encoders. More importantly, our approach seamlessly unlocks the combinations of the above capabilities, e.g., enabling in-context learning or instruction following with finetuned encoders. Experimental results across various language-only and vision-language benchmarks show that our model outperforms or is competitive with specialized models on finetuning, zero-shot generalization, and few-shot learning.
BibTeX
@misc{hao2022language,
  author = {Yaru Hao and Haoyu Song and Li Dong and Shaohan Huang and Zewen Chi and Wenhui Wang and Shuming Ma and Furu Wei},
  title = {{Language Models are General-Purpose Interfaces}},
  year = {2022},
  eprint = {2206.06336},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2206.06336},
}
Figure