Iteratively Prompt Pre-trained Language Models for Chain of Thought

Boshi Wang, Xiang Deng, Huan Sun

Introduction

Pre-trained Language Models (PLMs) like BART and GPT-3 serve as massive repositories of factual knowledge. However, they frequently struggle with complex, multi-step reasoning tasks. For instance, a model might correctly identify the father of a historical figure and separately know that father's birthplace, yet fail to answer a composite question like, "What is the birth place of Gwilym Lloyd George’s father?" This failure highlights a gap between stored knowledge and the ability to retrieve it in a structured "chain of thought."

The Problem: Static Probing and Fine-Tuning

Traditional methods for eliciting knowledge from PLMs have significant limitations. Standard Probing often results in "Unknown" responses for multi-hop queries because it attempts to bridge the reasoning gap in a single step. Conversely, Full Fine-tuning is computationally expensive and risks "catastrophic forgetting," where the model loses general knowledge while specializing in a specific task. Static Prompting (e.g., Prefix-Tuning) uses fixed tokens that cannot adapt as the reasoning process moves from one sub-fact to the next.

Comparison between standard probing and the proposed iterative prompting approach.

The iCAP Framework

To address these issues, researchers from Ohio State University introduced iCAP (iterative Context-Aware Prompter). Instead of a one-shot query, iCAP shepherds the PLM through a sequence of reasoning steps to recall facts from its own memory.

Architecture and Mechanism

The framework utilizes a large, frozen PLM (BART-large) as a static knowledge base and a smaller, trainable "Prompter" (RoBERTa-base). At each step $j$, the Prompter analyzes the original question $q$ and all previously recalled facts ($c_1, ..., c_{j-1}$) to generate a new, context-specific "soft prompt." This prompt steers the PLM to retrieve the next relevant piece of evidence.

Detailed architecture of the iCAP framework, showing the interaction between the Prompter and the frozen PLM blocks.

Key features of iCAP include:

  • Context-Awareness: Prompts evolve dynamically based on the current state of the reasoning chain.
  • Continuous Prompts: The system learns "virtual tokens" in the embedding space, which are more expressive and easier to optimize than natural language.
  • Parameter Efficiency: Only the small Prompter is trained, preserving the integrity of the large PLM's internal knowledge while avoiding the high costs of full model updates.

Experimental Evaluation

The framework was evaluated on three datasets requiring multi-step inference: 2WikiMultiHopQA, R4C, and Leap-of-Thought (LoT).

Key Results

  • Performance Boost: On the 2Wiki dataset, iCAP achieved an F1 score of 47.9, significantly outperforming static Prefix-Tuning (36.4) and Prompt-Tuning (30.2).
  • Closing the Gap: iCAP approached the performance of full PLM fine-tuning (50.9 F1) while keeping the main model's parameters untouched.
  • Faithfulness: Attention visualizations confirmed that the Prompter shifts its focus across different parts of the query as the reasoning steps progress (e.g., focusing on "father" first, then "birthplace").

Conclusion

iCAP demonstrates that complex reasoning in PLMs is often a retrieval problem rather than a lack of knowledge. By using a small, context-aware "librarian" to guide the retrieval process, we can unlock the reasoning potential of large models without the costs of extensive fine-tuning. Future work will explore scaling this approach to even larger models and handling noisier, real-world supervision.

Summary updated

Abstract

While Pre-trained Language Models (PLMs) internalize a great amount of world knowledge, they have been shown incapable of recalling these knowledge to solve tasks requiring complex & multi-step reasoning. Similar to how humans develop a "chain of thought" for these tasks, how can we equip PLMs with such abilities? In this work, we explore an iterative prompting framework, a new prompting paradigm which progressively elicits relevant knowledge from PLMs for multi-step inference. We identify key limitations of existing prompting methods, namely they are either restricted to queries with a single identifiable relation/predicate, or being agnostic to input contexts, which makes it difficult to capture variabilities across different inference steps. We propose an iterative context-aware prompter, which addresses these limitations by learning to dynamically synthesize prompts conditioned on the current step's contexts. Experiments on three datasets involving multi-step reasoning show the effectiveness of the iterative scheme and the context-aware prompter design.
BibTeX
@inproceedings{wang2022iteratively,
  author = {Boshi Wang and Xiang Deng and Huan Sun},
  title = {{Iteratively Prompt Pre-trained Language Models for Chain of Thought}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=2DFtNRvcAnU},
}
Figure