TL;DR
Chorus is a novel architecture that allows language models to share a single set of massive n-gram embedding tables across all layers, rather than maintaining redundant tables for each layer. By using layer-wise adaptation—rescaling and projecting shared embeddings based on a layer's specific hidden state—Chorus achieves the same performance as previous methods while using 87% less memory.
What Problem Does This Paper Address?
In the quest to scale language models (LMs), researchers often use n-gram embedding tables to add "cheap" capacity. These tables store learned vectors for short sequences (like "the cat" or "sat on the"), which are looked up and added to the model's internal representations. Because only a few rows are accessed per token, these tables can hold billions of parameters with almost zero additional computational cost.
However, existing designs like DeepSeek’s Engram or FactorEngram typically use a separate table for every layer that needs one. This creates two major issues:
- Memory Bloat: Storing unique tables for every layer is incredibly VRAM-intensive.
- Redundancy: Different layers are forced to learn the same n-gram patterns from scratch, preventing the model from sharing knowledge across its depth.
Simply sharing one table across all layers usually fails because early layers and late layers perform different computations and have different "preferences" for what information they need from an n-gram.
Methods
The authors propose Chorus, which centralizes n-gram knowledge into a single "refined embedding" that every layer can access and adapt.
- Shared Lookup & Refinement: The model looks up n-gram vectors (bigrams and trigrams) from a single set of hashed tables. These are combined with the standard token embedding and passed through a causal convolution to create a single refined embedding ($z_t$).
- Layer-wise Adaptation: This is the core innovation. Instead of adding $z_t$ directly to every layer, each layer "reads it in its own way" through two steps:
- Rescaling: The layer uses its current hidden state to compute a channel-wise gate, emphasizing the specific components of the n-gram vector that are relevant to that layer's current context.
- Projection: The rescaled vector is passed through a layer-specific linear projection to map it into the specific format the layer requires.
- Memory-Efficient Optimization: To handle tables with billions of parameters, the authors use Ember, a recently proposed optimizer. Ember factors the second-moment estimates of the gradients, drastically reducing the memory required to store optimizer states.
Datasets
The researchers evaluated Chorus across several configurations:
- ClimbMix: A dataset used for a 9-layer model (20.7M backbone) to test scaling across table sizes from 20M to 9.6B parameters.
- FineWeb: Used to validate results on a 9-layer (17.1M) and an 18-layer (133M) backbone.
- DCLM Evaluation Collection: A suite of 22 downstream tasks including Wikidata QA, LAMBADA, HellaSwag, and ARC to test factual recall and reasoning.
Key Insights and Ideas
- Knowledge is Universal, Interpretation is Local: The most significant "aha" moment is that layers want to share n-gram knowledge. The bottleneck wasn't the information itself, but the lack of a mechanism to let layers interpret that information differently.
- Projections are Cheaper than Tables: A small projection matrix per layer is orders of magnitude smaller than a full n-gram table. This trade-off allows for massive parameter counts with a tiny memory footprint.
- The Optimizer is the Real Memory Hog: For billion-parameter tables, the weights aren't the only problem—the optimizer states (like AdamW's moments) are. By pairing Chorus with the Ember optimizer, the authors reduced optimizer memory by over 99%.
- Token Tables Still Matter: Ablation studies showed that keeping a dedicated token-level table ($U$) is crucial, even when high-order n-gram tables are present, likely because it provides a stable baseline for individual token identities.
Key Results
Chorus consistently outperformed all baselines (Engram, TN-gram, FactorEngram, and X-gram) across every scale and dataset tested.
- 87% Parameter Efficiency: A Chorus model with 1.2B table parameters predicted text better than Engram or FactorEngram models using 9.6B parameters.
- Scaling Superiority: As shown in Figure 1, Chorus maintains a significant lead in validation bits per byte (BPB) as table size increases.
- Downstream Performance: On the 22-task DCLM suite, Chorus achieved the highest mean score, showing particular strength in factual recall tasks like Wikidata QA (25.8% accuracy vs. ~18% for baselines).
- VRAM Savings: For a 1.2B parameter table, using Ember reduced the optimizer state from 4.47 GiB (AdamW) to just 35.7 MiB.
Limitations and Future Work
- Model Scale: The experiments were conducted on relatively small "backbone" models (up to 133M parameters). While the n-gram tables were large, it remains to be seen how Chorus scales with 7B+ parameter LLMs.
- Task Specificity: The gains were most pronounced in natural language prediction and factual recall. Reasoning-heavy tasks (like Dyck languages or CommonsenseQA) showed smaller, less statistically significant improvements.
- N-gram Order: The study primarily used up to 3-grams. The authors noted that larger tables might benefit from even longer contexts (4-grams or 5-grams), which warrants further investigation.
Summary updated