RECKONING: Reasoning through Dynamic Knowledge Encoding

Zeming Chen, Gail Weiss, Eric Mitchell, Asli Celikyilmaz, Antoine Bosselut

Introduction

RECKONING (REasoning through dynamiC KnOwledge eNcodING) is a novel bi-level learning framework that moves beyond traditional in-context reasoning. Instead of simply reading facts from a prompt, RECKONING "folds" new facts directly into a language model’s weights via rapid gradient updates before asking a question. This approach makes models significantly more robust to irrelevant "distractor" information and allows them to generalize to more complex reasoning chains than standard prompting methods.

The Problem with In-Context Reasoning

Most large language models (LLMs) perform reasoning using In-Context Reasoning (ICR): a set of facts and a question are provided in the prompt, and the model processes them in a single forward pass. However, ICR suffers from a major "distractor" problem. If the context contains facts that are irrelevant to the specific question—even if they are true facts about the world—the model’s performance often degrades significantly.

Interestingly, LLMs are quite adept at ignoring irrelevant information stored in their parametric knowledge (the weights they were pre-trained on). The authors pose a compelling question: What if we treat new contextual information like parametric knowledge? Instead of just reading the facts, the model could "memorize" them through a quick update to its weights before it attempts to reason.

Methods: A Bi-Level Optimization Approach

To achieve this, the authors introduce RECKONING, a bi-level optimization algorithm inspired by Meta-Learning (specifically MAML). The process works in two nested loops:

Bi-level optimization framework of RECKONING
  1. The Inner Loop (Knowledge Encoding): When presented with a set of facts, the model performs a few rapid gradient descent steps using a Causal Language Modeling (CLM) loss. This effectively "folds" the new facts into a temporary copy of the model's parameters.
  2. The Outer Loop (Reasoning Optimization): During training, the "meta-parameters" are optimized so that, after the inner loop update, the model is highly successful at answering reasoning questions related to those facts.

To make this work effectively, the authors use a Multi-Task Objective. The model isn't just trained to give the right answer; it’s also trained to be able to reproduce the facts it just "memorized." They also implement a Dynamic Learning Rate, where the model learns a specific, optimal learning rate for every layer and every step of the inner loop.

Datasets

The researchers evaluated RECKONING on three diverse multi-hop reasoning benchmarks:

  • ProofWriter: A synthetic dataset requiring logical deduction over natural language rules and facts.
  • CLUTRR-SG: A benchmark focused on family relationship reasoning, specifically testing "systematic generalization" to longer reasoning chains.
  • FOLIO: A real-world dataset featuring complex first-order logic problems written by expert annotators.

Key Insights

  • Weights are more robust than Windows: The core insight is that the transformer architecture is naturally better at filtering noise when that noise is embedded in its weights rather than sitting in its attention context window.
  • Amortized Efficiency: While performing gradient updates at inference time sounds slow, it’s actually a win for "multi-query" scenarios. If you have 1,000 facts and 20 questions about them, RECKONING encodes the facts once and then runs 20 fast forward passes. ICR would have to process all 1,000 facts in the prompt for every single question.
  • Extrapolation over Interpolation: RECKONING shows a superior ability to handle "out-of-distribution" complexity. By encoding the knowledge into weights, the model maintains much higher accuracy on longer, unseen reasoning chains compared to standard context window approaches.

Key Results

  • Accuracy Boost: RECKONING outperformed fine-tuned ICR baselines by up to 4.5% across the benchmarks.
  • Distractor Robustness: As the number of irrelevant facts increases, RECKONING maintains its performance much better than standard methods. In ProofWriter tests with distractor facts, RECKONING achieved 82.5% accuracy, while the standard ICR baseline dropped to 70.9%.
Performance comparison with varying numbers of distractors
  • Beating the Giants: Despite being based on the relatively small GPT-2 architecture (124M and 1.5B versions), RECKONING outperformed much larger models like GPT-3.5 and ChatGPT (in zero-shot and 8-shot settings) on the FOLIO and ProofWriter benchmarks.
  • Efficiency: For 18+ questions about the same knowledge set, RECKONING becomes computationally faster than standard ICR because it avoids redundant processing of the long context.

Limitations and Future Work

  • Inference Latency: For a single question, RECKONING is slower than a standard forward pass because of the required gradient steps.
  • Scaling to Massive Models: The experiments went up to 1.5B parameters. It remains to be seen how this bi-level optimization scales to 70B+ parameter models where full gradient updates are extremely expensive (though Parameter-Efficient Fine-Tuning like LoRA is a suggested solution).
  • Knowledge Capacity: There is likely a limit to how much information can be "folded" into the weights before the model experiences interference or "forgetting" of its base capabilities. Exploring these boundaries is a critical next step.

Summary updated

Abstract

Recent studies on transformer-based language models show that they can answer questions by reasoning over knowledge provided as part of the context (i.e., in-context reasoning). However, since the available knowledge is often not filtered for a particular question, in-context reasoning can be sensitive to distractor facts, additional content that is irrelevant to a question but that may be relevant for a different question (i.e., not necessarily random noise). In these situations, the model fails to distinguish the necessary knowledge to answer the question, leading to spurious reasoning and degraded performance. This reasoning failure contrasts with the model’s apparent ability to distinguish its contextual knowledge from all the knowledge it has memorized during pre-training. Following this observation, we propose teaching the model to reason more robustly by folding the provided contextual knowledge into the model’s parameters before presenting it with a question. Our method, RECKONING, is a bi-level learning algorithm that teaches language models to reason by updating their parametric knowledge through back-propagation, allowing them to answer questions using the updated parameters. During training, the inner loop rapidly adapts a copy of the model weights to encode contextual knowledge into its parameters. In the outer loop, the model learns to use the updated weights to reproduce and answer reasoning questions about the memorized knowledge. Our experiments on three diverse multi-hop reasoning datasets show that RECKONING’s performance improves over the in-context reasoning baseline (by up to 4.5%). We also find that compared to in-context reasoning, RECKONING generalizes better to longer reasoning chains unseen during training, is more robust to distractors in the context, and is computationally more efficient when multiple questions are asked about the same knowledge.
BibTeX
@inproceedings{chen2023reckoning,
  author = {Zeming Chen and Gail Weiss and Eric Mitchell and Asli Celikyilmaz and Antoine Bosselut},
  title = {{RECKONING: Reasoning through Dynamic Knowledge Encoding}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=dUAcAtCuKk},
}
Figure