Abstract
Large Language Models (LLMs) exhibit emergent reasoning capabilities, but smaller models often struggle to generalize, instead memorizing specific reasoning paths during knowledge distillation. We present PRADA (PRompt-Assisted Domain-Adversarial fine-tuning), a framework that combines prompt-based learning with domain-adversarial training to bridge this generalization gap. PRADA enables student models, even those 250x smaller than their teachers, to achieve robust Chain-of-Thought (CoT) reasoning across diverse, unseen domains.
Introduction: The Memorization Trap
Chain-of-Thought (CoT) reasoning—the ability to decompose complex problems into step-by-step logical sequences—is a hallmark of massive models like GPT-4. While Knowledge Distillation (KD) can transfer these capabilities to smaller language models (SLMs), these smaller counterparts often fall into a "memorization trap." Rather than internalizing the underlying logic, SLMs tend to rote-memorize specific reasoning patterns from the teacher.
As shown in the figure above, the "generalization gap" between source and target domain accuracy becomes more pronounced as model size decreases. This lack of robustness makes SLMs unreliable for real-world applications where data distributions frequently shift.
The PRADA Framework
To address this, we propose PRADA, a three-stage pipeline designed to extract domain-invariant reasoning features.
1. Diverse Reasoning Generation
A large teacher model (e.g., GPT-3 175B) is prompted using Zero-Shot-CoT to generate multiple, diverse reasoning paths for each problem. This provides the student with a variety of logical "templates," reducing the likelihood of memorizing a single path.
2. Prompt Learning (P-Tuning)
An adapter layer is inserted into the student model. This layer utilizes continuous prompt embeddings to refine domain-agnostic knowledge. This step stabilizes the learning process before the model undergoes more intensive adversarial training.
3. Domain-Adversarial Fine-Tuning
The core of PRADA is the integration of a domain classifier and a Gradient Reversal Layer (GRL). During training, the model is tasked with solving the reasoning problem while the domain classifier attempts to identify the source domain of the input. The GRL forces the model to learn features that make domains indistinguishable, effectively stripping away domain-specific "noise" and focusing on universal logical structures.
Experimental Evaluation
PRADA was evaluated across 12 datasets covering arithmetic, symbolic, and commonsense reasoning. Using GPT-3 (175B) as the teacher, we tested student models including Flan-T5-Large and Llama-3-8B.
Key Results
- Significant Gains: On the SVAMP dataset, PRADA improved Llama-3-8B's accuracy by over 10% compared to pretrained baselines.
- Symbolic Reasoning: In the "Coin Flip" task, PRADA enabled Llama-3 to achieve 96.99% accuracy, far surpassing standard fine-tuning methods.
- Efficiency: Student models up to 250x smaller than the teacher retained a high percentage of reasoning capability, demonstrating the framework's effectiveness for edge deployment.
Conclusion
PRADA successfully mitigates the memorization-to-generalization transition in smaller models. By leveraging adversarial logic to isolate pure reasoning structures, we can deploy efficient, robust models that "think" rather than just repeat. Future work will explore scaling PRADA to sub-1B parameter models and multi-modal reasoning tasks.
Summary updated