Chorus: Shared N-gram Embeddings with Layer-wise Adaptation

Stephen Chung, Wenyu Du

TL;DR

Chorus is a novel architecture that allows language models to share a single set of massive n-gram embedding tables across all layers, rather than maintaining redundant tables for each layer. By using layer-wise adaptation—rescaling and projecting shared embeddings based on a layer's specific hidden state—Chorus achieves the same performance as previous methods while using 87% less memory.

What Problem Does This Paper Address?

In the quest to scale language models (LMs), researchers often use n-gram embedding tables to add "cheap" capacity. These tables store learned vectors for short sequences (like "the cat" or "sat on the"), which are looked up and added to the model's internal representations. Because only a few rows are accessed per token, these tables can hold billions of parameters with almost zero additional computational cost.

However, existing designs like DeepSeek’s Engram or FactorEngram typically use a separate table for every layer that needs one. This creates two major issues:

  1. Memory Bloat: Storing unique tables for every layer is incredibly VRAM-intensive.
  2. Redundancy: Different layers are forced to learn the same n-gram patterns from scratch, preventing the model from sharing knowledge across its depth.

Simply sharing one table across all layers usually fails because early layers and late layers perform different computations and have different "preferences" for what information they need from an n-gram.

Methods

The authors propose Chorus, which centralizes n-gram knowledge into a single "refined embedding" that every layer can access and adapt.

  1. Shared Lookup & Refinement: The model looks up n-gram vectors (bigrams and trigrams) from a single set of hashed tables. These are combined with the standard token embedding and passed through a causal convolution to create a single refined embedding ($z_t$).
  2. Layer-wise Adaptation: This is the core innovation. Instead of adding $z_t$ directly to every layer, each layer "reads it in its own way" through two steps:
  3. Rescaling: The layer uses its current hidden state to compute a channel-wise gate, emphasizing the specific components of the n-gram vector that are relevant to that layer's current context.
  4. Projection: The rescaled vector is passed through a layer-specific linear projection to map it into the specific format the layer requires.
  5. Memory-Efficient Optimization: To handle tables with billions of parameters, the authors use Ember, a recently proposed optimizer. Ember factors the second-moment estimates of the gradients, drastically reducing the memory required to store optimizer states.
Figure 2: Existing n-gram embedding tables and Chorus. (a) Engram, TN-gram and FactorEngram keep a separate n-gram table for every place that reads one, including the input. (b) Chorus looks up one set of tables once and reuses the result at the input and in every other layer. Each layer rescales the refined embedding using its own hidden state (dashed arrows) and maps it through its own projection.

Datasets

The researchers evaluated Chorus across several configurations:

  • ClimbMix: A dataset used for a 9-layer model (20.7M backbone) to test scaling across table sizes from 20M to 9.6B parameters.
  • FineWeb: Used to validate results on a 9-layer (17.1M) and an 18-layer (133M) backbone.
  • DCLM Evaluation Collection: A suite of 22 downstream tasks including Wikidata QA, LAMBADA, HellaSwag, and ARC to test factual recall and reasoning.

Key Insights and Ideas

  • Knowledge is Universal, Interpretation is Local: The most significant "aha" moment is that layers want to share n-gram knowledge. The bottleneck wasn't the information itself, but the lack of a mechanism to let layers interpret that information differently.
  • Projections are Cheaper than Tables: A small projection matrix per layer is orders of magnitude smaller than a full n-gram table. This trade-off allows for massive parameter counts with a tiny memory footprint.
  • The Optimizer is the Real Memory Hog: For billion-parameter tables, the weights aren't the only problem—the optimizer states (like AdamW's moments) are. By pairing Chorus with the Ember optimizer, the authors reduced optimizer memory by over 99%.
  • Token Tables Still Matter: Ablation studies showed that keeping a dedicated token-level table ($U$) is crucial, even when high-order n-gram tables are present, likely because it provides a stable baseline for individual token identities.

Key Results

Chorus consistently outperformed all baselines (Engram, TN-gram, FactorEngram, and X-gram) across every scale and dataset tested.

  • 87% Parameter Efficiency: A Chorus model with 1.2B table parameters predicted text better than Engram or FactorEngram models using 9.6B parameters.
  • Scaling Superiority: As shown in Figure 1, Chorus maintains a significant lead in validation bits per byte (BPB) as table size increases.
  • Downstream Performance: On the 22-task DCLM suite, Chorus achieved the highest mean score, showing particular strength in factual recall tasks like Wikidata QA (25.8% accuracy vs. ~18% for baselines).
  • VRAM Savings: For a 1.2B parameter table, using Ember reduced the optimizer state from 4.47 GiB (AdamW) to just 35.7 MiB.
Figure 1: Scaling curves with respect to table size. ClimbMix, nine-layer model (20.7M backbone), ten table sizes. Validation bits per byte (lower is better) against table size. Curves show Chorus, Engram and FactorEngram; isolated points show OE, TN-gram and two X-gram configurations. One seed per point.

Limitations and Future Work

  • Model Scale: The experiments were conducted on relatively small "backbone" models (up to 133M parameters). While the n-gram tables were large, it remains to be seen how Chorus scales with 7B+ parameter LLMs.
  • Task Specificity: The gains were most pronounced in natural language prediction and factual recall. Reasoning-heavy tasks (like Dyck languages or CommonsenseQA) showed smaller, less statistically significant improvements.
  • N-gram Order: The study primarily used up to 3-grams. The authors noted that larger tables might benefit from even longer contexts (4-grams or 5-grams), which warrants further investigation.

Summary updated

Abstract

Language models can gain capacity at little computational cost from n-gram embedding tables: large tables of learned vectors for short sequences of consecutive tokens, which are looked up and added to the model's input or to its intermediate layers. Recent designs commonly use a separate n-gram embedding table for each layer, which makes the tables costly in memory and prevents layers from sharing what they learn. Sharing a single table, however, gives every layer the same embedding and leaves no room for layer-wise preferences, and it performs poorly in practice. We propose Chorus, in which one set of n-gram embedding tables serves the whole network and every layer reads it in its own way. Each layer maps the refined embedding through its own projection layer, after rescaling its components by factors computed from its current hidden state. Chorus matches the quality of existing n-gram methods with 87% smaller tables: with 1.2B table parameters it predicts text better than FactorEngram and DeepSeek's Engram do with 9.6B. It outperforms Engram, TN-gram and FactorEngram in every setting where we compared them, across table sizes from 20M to 9.6B parameters. We also show that the recently proposed memory-efficient optimizer Ember works well with Chorus, saving 99.2% of table-optimizer memory compared with AdamW without degrading training performance. These results enable further scaling of n-gram embedding tables along the table-size axis by reducing memory use and allowing shared knowledge to be reused across layers.
BibTeX
@misc{chung2026chorus,
  author = {Stephen Chung and Wenyu Du},
  title = {{Chorus: Shared N-gram Embeddings with Layer-wise Adaptation}},
  year = {2026},
  eprint = {2611.23386},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23386},
}
Figure