Abstract
This paper introduces CEKFNN (Conditional Extended Kalman Filter based Neural Network), a novel framework that conceptualizes prompt learning as a state-estimation problem. By employing an Extended Kalman Filter (EKF) to dynamically generate prompt tokens conditioned on sentence context and sequence history, CEKFNN achieves performance comparable to or exceeding full fine-tuning while updating only a fraction of the model's parameters. The approach addresses the limitations of static prompts by making them context-aware and temporally fluid.
Introduction
Pre-trained Language Models (PLMs) such as BERT, GPT, and T5 have revolutionized NLP, but their massive scale makes full fine-tuning computationally prohibitive. Prompt learning emerged as a parameter-efficient alternative, using small sets of "trigger" tokens to guide model behavior. However, traditional methods typically employ static prompts that ignore the semantic nuances of individual sentences and the relationships between consecutive inputs. This paper argues that for prompts to truly unlock the knowledge within PLMs, they must be dynamic and context-aware.
The CEKFNN Framework
The authors treat the prompt learning process through the lens of control theory, specifically the Extended Kalman Filter (EKF). In this paradigm, prompt tokens are modeled as latent (hidden) variables that evolve over time, while task labels serve as the observed variables.
System Components
- Initial Model: Establishes the starting state for prompt tokens.
- Transition Function: A single-layer Transformer encoder that updates prompt tokens between steps, conditioned on the current sentence's embeddings. This allows the model to maintain a "semantic thread" across related sentences.
- Emission Function: An adapter-based PLM (utilizing LoRA) that processes the dynamic prompt tokens and sentence embeddings to predict the final label.
Alternate Training Procedure
To optimize this system, the researchers developed a two-phase "alternate training" strategy. In Phase I, the Prompt Models (PMs) generate tokens to "warm up" the PLM's emission function. In Phase II, a mutual enhancement loop is established: PMs are trained using feedback from the fixed PLM, followed by the PLM (via its adapters) being retrained using the optimized prompts. This iterative process ensures that the prompts and the model are perfectly synchronized.
Experimental Results
CEKFNN was evaluated against standard benchmarks including GLUE and SuperGLUE across various model architectures (BERT, GPT-2, and T5-3B).
Key Findings
- Efficiency vs. Performance: On the BERT-Base CoLA task, CEKFNN achieved a score of 60.5, outperforming P-Tuning (47.5) and even slightly exceeding full fine-tuning (59.6), despite using approximately 90% fewer trainable parameters.
- Superiority in NLI: For GPT-2 Medium on the WNLI task, CEKFNN reached a score of 70.8, a significant improvement over the 56.5 achieved by standard fine-tuning.
- Scalability: When applied to the T5-3B model, CEKFNN outperformed other adapter methods on challenging SuperGLUE tasks like CB and COPA, demonstrating its effectiveness for extremely large-scale models.
Insights and Discussion
The core innovation of CEKFNN is viewing a prompt as a "hidden state" of a dynamical system rather than a static string. This allows for contextual fluidity, where the model adjusts its internal state based on specific inputs and historical context. Ablation studies confirmed that the iterative alternate training loop is essential; without it, performance on linguistically complex tasks like CoLA drops significantly.
While highly effective, the framework's architectural complexity is higher than simple prompt concatenation. Future research directions include extending this EKF-based approach to Natural Language Generation (NLG) and other domains such as Computer Vision or signal processing.
Summary updated