Memory Augmented Language Models through Mixture of Word Experts

Cícero Nogueira dos Santos, James Lee-Thorp, Isaac Noble, Chung-Ching Chang, David C. UthusNAACL 2024 · DOI

TL;DR

This paper introduces Mixture-of-Word-Experts (MoWE), a novel neural network architecture that combines the efficiency of sparse Mixture-of-Experts (MoE) models with the knowledge-retrieval capabilities of memory-augmented models. By using a large, knowledge-rich vocabulary to route tokens to tens of thousands of word-specific experts, MoWE achieves significant performance gains on knowledge-intensive tasks while remaining computationally efficient.

What Problem Does This Paper Address?

Large language models (LLMs) have shown remarkable improvements in quality, particularly on knowledge-intensive tasks. However, scaling up these models by increasing parameter count also leads to a proportional increase in computational cost for training and inference. This presents a trade-off between model quality and efficiency. The paper aims to decouple learning capacity from computational cost (FLOPs) by developing a sparse model that can effectively memorize and retrieve world knowledge, similar to how memory-augmented models operate, but without the need for complex external memory search mechanisms.

Methods

The core innovation is the Mixture-of-Word-Experts (MoWE) architecture. MoWE replaces the Feed-Forward Network (FFN) layers in a subset of Transformer blocks with a sparse layer of experts. Unlike traditional MoE models that use a learned routing function to assign tokens to a limited number of experts (typically 32-128), MoWE employs two key principles:

  1. Massive Number of Experts: MoWE utilizes tens of thousands of experts, significantly more than standard MoE models.
  2. Knowledge-Rich Routing Vocabulary: A large, fixed vocabulary derived from knowledge bases (like Wikidata) is used to route tokens. Each token is associated with a "routing id" which maps to a specific "word-specific" expert. This creates a "sparse memory" where experts specialize in particular words or concepts.

To handle the challenges of managing such a large number of experts, MoWE introduces a hierarchical routing mechanism. Tokens are first routed to frequency buckets, then to expert blocks within those buckets, and finally to individual experts. This strategy helps manage communication overhead.

Datasets

  • Pretraining: The C4 dataset (version 2.2.0) was used for pretraining.
  • Downstream Tasks:
    • Knowledge-Intensive Tasks: TriviaQA, WebQuestions, Natural Questions.
  • General NLP Tasks: FEVER (claim verification), SuperGLUE.
  • Routing Vocabulary Construction: A Wikidata dump was used to extract entity and relation names, which were then processed to form the knowledge-rich routing vocabulary.

Key Insights and Ideas

  • Word Experts as Sparse Memory: The paper posits that a large number of word-specific experts, guided by a knowledge-rich vocabulary, effectively act as a sparse, integrated memory within the language model. This allows for efficient storage and retrieval of factual knowledge.
  • Fixed Routing is Sufficient: Contrary to learned routing in standard MoE, MoWE's fixed routing based on a knowledge-rich vocabulary proves effective, simplifying training and avoiding the need for complex routing function learning.
  • Specialization through Scale: The sheer number of experts enables fine-grained specialization, allowing individual experts to focus on specific words or concepts, which is crucial for knowledge-intensive tasks.
  • Efficiency Gains: MoWE achieves comparable or better performance than dense models with significantly fewer FLOPs, demonstrating a successful decoupling of capacity and computation.

Key Results

  • Performance on Knowledge-Intensive Tasks: MoWE-Base outperforms T5.1.1-XL on knowledge-intensive tasks (TriviaQA, WebQuestions, FEVER) while being 4.3x faster to train. MoWE-Large outperforms T5.1.1-XXL, achieving a 6.6x training speedup.
  • TriviaQA Improvement: MoWE-Base shows a 15.2-point improvement in exact match on TriviaQA compared to T5.1.1-Base (a 62.8% improvement).
  • Comparison with Regular MoEs: MoWE models outperform or match state-of-the-art sparse MoE models like ST-MoE-L on knowledge-intensive tasks, while achieving comparable performance on SuperGLUE.
  • Memory Integration: Experiments show that skipping word experts in MoWE significantly degrades performance on TriviaQA, highlighting the critical role of these experts in storing and retrieving knowledge.
  • Training Speed: MoWE models achieve substantial training speedups (e.g., 4.3x for MoWE-Base vs. T5.1.1-XL).

Limitations and Future Work

  • Computational Budget: The experiments were limited to MoWE-Large (45.5B parameters), and testing larger configurations (MoWE-XL) was not feasible due to computational constraints. The authors expect the performance trend to continue.
  • Expert Properties: The paper acknowledges that the properties and relationships between word experts are not deeply explored. Future work could investigate similarity metrics between experts or how to find relevant nearest neighbors.
  • Decoder-Only Models: MoWE layers were not explored in decoder-only models, focusing instead on T5-based encoder-decoder architectures for comparison.
  • Domain Specificity: Current experiments are limited to NLP tasks and English datasets. Future work could extend MoWE to other modalities and languages.
  • Routing Vocabulary Optimization: While effective, the routing vocabulary construction could be further improved.
  • Future Directions: The authors suggest exploring the properties of word experts, integrating MoWE into decoder-only models, and extending it to other modalities and languages.

Summary updated

BibTeX
@inproceedings{santos2024memory,
  author = {Cícero Nogueira dos Santos and James Lee-Thorp and Isaac Noble and Chung-Ching Chang and David C. Uthus},
  title = {{Memory Augmented Language Models through Mixture of Word Experts}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.249},
}
Figure