N-Gram Induction Heads for In-Context RL: Improving Stability and Reducing Data Needs

Ilya Zisman, Alexander Nikulin, Viacheslav Sinii, Denis Tarasov, Lyubaykin Nikita, Andrei Polubarov, Igor Kiselev, Vladislav KurenkovSubmission to ICLR 2026 · Rejected

TL;DR

By hardcoding n-gram induction heads into Transformer architectures, researchers have significantly boosted the efficiency of In-Context Reinforcement Learning (ICRL). This modification allows agents to generalize to new tasks with up to 27x less data and makes the training process far less sensitive to hyperparameter tuning compared to standard Algorithm Distillation.

What Problem Does This Paper Address?

In-Context Reinforcement Learning (ICRL) is a fascinating frontier where a model (usually a Transformer) learns to solve a new task during inference simply by looking at a few examples of interactions, without any gradient updates to its weights. The current gold standard, Algorithm Distillation (AD), works by training a model on "learning histories"—the step-by-step progress of a traditional RL algorithm.

However, ICRL has two major "pain points":

  1. Data Hunger: It requires massive, carefully curated datasets of thousands of RL training runs to learn how to "learn."
  2. Instability: Transformers suffer from a simplicity bias. They tend to learn simple patterns (like uni-grams) first, and the complex "induction heads" required for sophisticated in-context reasoning take a long time to emerge. This makes training unstable and extremely sensitive to hyperparameters; a model might suddenly lose its in-context abilities mid-training.

Methods

The authors propose a model-centric fix: instead of waiting for the Transformer to eventually learn how to track repeating patterns, why not just build that capability into the architecture?

  • N-Gram Attention Layer: They integrate a specialized layer that calculates an attention pattern based on exact matches of sequences (n-grams). If the model sees a sequence of observations and actions it has seen before, the n-gram head forces the model to attend to what happened next in those previous instances.
  • Drop-in Replacement: This n-gram layer acts as a drop-in component alongside standard multi-head attention. It provides a strong inductive bias that helps the model recognize and exploit repeating structures in the environment's dynamics.
  • Visual Quantization: To make this work for pixel-based environments (where two images are rarely exactly the same), the authors use Vector Quantization (VQ). They pre-train a ResNet-based VQ-VAE to map images into a $4 \times 4$ matrix of discrete indices. The n-gram matching then operates on these discrete "visual tokens."

Datasets

The researchers evaluated their approach on several benchmarks designed to test adaptation and memory:

  • Dark Room: A $9 \times 9$ grid-world where the agent must find a hidden goal.
  • Dark Key-to-Door: A more complex POMDP (Partially Observable Markov Decision Process) where the agent must find a key in one phase to unlock a door in another. This requires long-term memory.
  • Miniworld (Dark & Key-to-Door): 3D, first-person pixel-based versions of the above environments, used to test the model's ability to handle visual observations.

Key Insights and Ideas

  • Bypassing Simplicity Bias: By hardcoding n-gram heads, the model doesn't have to "discover" the concept of looking at past similar states. This jump-starts the learning process, allowing the model to focus on higher-level strategy rather than basic pattern matching.
  • Hyperparameter "Easy Mode": One of the most striking findings is how much easier the model is to tune. In random hyperparameter searches, the n-gram version reached near-optimal performance in a fraction of the attempts required by the baseline.
  • Generalization from Scarcity: The inductive bias is so strong that the model can generalize to unseen tasks even when the training set is significantly restricted. It effectively "overfits" less on the specific training tasks and learns the process of reinforcement learning more robustly.

Key Results

  • Massive Data Savings: In the Key-to-Door environment, the proposed method achieved performance comparable to the baseline while using 27x fewer transitions in the training data.
  • Search Efficiency: For the Dark Room task with 1,000 learning histories, the n-gram model found optimal hyperparameters in roughly 20 assignments, whereas the baseline required over 400 to reach the same level.
  • Visual Success: The quantization strategy worked. Even in 3D environments with visual noise, the n-gram heads successfully identified repeating states, allowing the agent to adapt in-context as effectively as it did in discrete grid-worlds.

Limitations and Future Work

While the results are impressive, the authors note a few areas for improvement:

  • Environment Complexity: The tests were conducted in relatively controlled environments (grid-worlds and Miniworld). Testing on more diverse, high-dimensional benchmarks like XLand-Minigrid or Meta-World is a necessary next step.
  • Continuous States: The current n-gram matching relies on discrete tokens. Adapting this logic to work with continuous proprioceptive states (like joint angles in robotics) remains an open challenge.
  • Quantization Dependency: For visual tasks, the method relies on a pre-trained VQ model. Integrating the quantization and the ICRL training into a single end-to-end process could further improve efficiency.

Summary updated

Abstract

In-context learning allows models like transformers to adapt to new tasks from a few examples without updating their weights, a desirable trait for reinforcement learning (RL). However, existing in-context RL methods, such as Algorithm Distillation (AD), demand large, carefully curated datasets and can be unstable and costly to train due to the transient nature of in-context learning abilities. In this work, we integrated the n-gram induction heads into transformers for in-context RL. By incorporating these n-gram attention patterns, we considerably reduced the amount of data required for generalization and eased the training process by making models less sensitive to hyperparameters. Our approach matches, and in some cases surpasses, the performance of AD in both grid-world and pixel-based environments, suggesting that n-gram induction heads could improve the efficiency of in-context RL.
BibTeX
@inproceedings{zisman2024n,
  author = {Ilya Zisman and Alexander Nikulin and Viacheslav Sinii and Denis Tarasov and Lyubaykin Nikita and Andrei Polubarov and Igor Kiselev and Vladislav Kurenkov},
  title = {{N-Gram Induction Heads for In-Context RL: Improving Stability and Reducing Data Needs}},
  booktitle = {AFM @ NeurIPS},
  year = {2024},
  url = {https://openreview.net/forum?id=0f5XmCs8VR},
}
Figure