Introduction
Pre-trained Language Models (PLMs) like BART and GPT-3 serve as massive repositories of factual knowledge. However, they frequently struggle with complex, multi-step reasoning tasks. For instance, a model might correctly identify the father of a historical figure and separately know that father's birthplace, yet fail to answer a composite question like, "What is the birth place of Gwilym Lloyd George’s father?" This failure highlights a gap between stored knowledge and the ability to retrieve it in a structured "chain of thought."
The Problem: Static Probing and Fine-Tuning
Traditional methods for eliciting knowledge from PLMs have significant limitations. Standard Probing often results in "Unknown" responses for multi-hop queries because it attempts to bridge the reasoning gap in a single step. Conversely, Full Fine-tuning is computationally expensive and risks "catastrophic forgetting," where the model loses general knowledge while specializing in a specific task. Static Prompting (e.g., Prefix-Tuning) uses fixed tokens that cannot adapt as the reasoning process moves from one sub-fact to the next.
The iCAP Framework
To address these issues, researchers from Ohio State University introduced iCAP (iterative Context-Aware Prompter). Instead of a one-shot query, iCAP shepherds the PLM through a sequence of reasoning steps to recall facts from its own memory.
Architecture and Mechanism
The framework utilizes a large, frozen PLM (BART-large) as a static knowledge base and a smaller, trainable "Prompter" (RoBERTa-base). At each step $j$, the Prompter analyzes the original question $q$ and all previously recalled facts ($c_1, ..., c_{j-1}$) to generate a new, context-specific "soft prompt." This prompt steers the PLM to retrieve the next relevant piece of evidence.
Key features of iCAP include:
- Context-Awareness: Prompts evolve dynamically based on the current state of the reasoning chain.
- Continuous Prompts: The system learns "virtual tokens" in the embedding space, which are more expressive and easier to optimize than natural language.
- Parameter Efficiency: Only the small Prompter is trained, preserving the integrity of the large PLM's internal knowledge while avoiding the high costs of full model updates.
Experimental Evaluation
The framework was evaluated on three datasets requiring multi-step inference: 2WikiMultiHopQA, R4C, and Leap-of-Thought (LoT).
Key Results
- Performance Boost: On the 2Wiki dataset, iCAP achieved an F1 score of 47.9, significantly outperforming static Prefix-Tuning (36.4) and Prompt-Tuning (30.2).
- Closing the Gap: iCAP approached the performance of full PLM fine-tuning (50.9 F1) while keeping the main model's parameters untouched.
- Faithfulness: Attention visualizations confirmed that the Prompter shifts its focus across different parts of the query as the reasoning steps progress (e.g., focusing on "father" first, then "birthplace").
Conclusion
iCAP demonstrates that complex reasoning in PLMs is often a retrieval problem rather than a lack of knowledge. By using a small, context-aware "librarian" to guide the retrieval process, we can unlock the reasoning potential of large models without the costs of extensive fine-tuning. Future work will explore scaling this approach to even larger models and handling noisier, real-world supervision.
Summary updated