TL;DR
As Small Language Models (SLMs) are increasingly deployed on edge devices for personalized applications, they face a high risk of catastrophic forgetting during sequential updates. This paper introduces a lightweight, checkpoint-level monitoring protocol that uses a fixed reference set and KL divergence to detect internal model instability before it manifests as a total collapse in task performance.
What Problem Does This Paper Address?
Deploying Small Language Models (SLMs, typically <1B parameters) directly on edge devices like laptops and smartphones enables private, low-latency, and highly tailored user experiences. However, to personalize effectively, these models must adapt over time to a continuous stream of user- or task-specific data. This setup triggers a classic continual learning dilemma: catastrophic forgetting, where learning new information degrades the model's previously acquired knowledge or foundational capabilities.
While existing benchmarks evaluate a model's final performance after training, they lack mechanisms to monitor stability during the sequential adaptation process. A model might successfully improve on a newly introduced task while quietly drifting away from its original alignment, safety boundaries, or general reasoning capabilities. The authors address this gap by developing a practical, low-cost diagnostic protocol to monitor model health in real-time during sequential edge personalization.
Methods
The authors propose a checkpoint-level stability monitoring protocol. Instead of training separate adapters for every task or only evaluating the final model, they train a single cumulative Low-Rank Adaptation (LoRA) adapter sequentially across a series of tasks.
After each adaptation stage, the resulting model checkpoint is saved and evaluated against a checkpoint-by-task matrix. Crucially, the protocol introduces a fixed reference set ($R$) that is completely isolated from training. By analyzing the model's next-token predictive distributions on this reference set across different checkpoints, they compute three lightweight distributional diagnostics:
- KL Divergence: Measures how much the adapted model's token distribution has drifted from the initial base model ($\theta_0$).
- $\Delta$ Entropy: Tracks changes in model uncertainty and confidence.
- $\Delta$ Margin: Measures the log-probability separation between the top two most likely tokens to determine if the model is becoming less decisive.
Datasets
To simulate a realistic, data-constrained edge personalization scenario, the authors subsampled three distinct datasets from the TRACE benchmark, limiting each to just 500 training examples:
- ScienceQA: A question-and-answer dataset drawn from elementary and high school science curricula.
- FOMC: A financial sentiment classification task designed to categorize central bank communications as hawkish, dovish, or neutral.
- NumGLUE-CM: A mathematical dataset focused on arithmetic reasoning problems.
The specific characteristics of these evaluation datasets are outlined below: Datasets used in the TRACE-style continual learning evaluation
Key Insights and Ideas
- KL Divergence as an Early Warning Signal: Internal distributional drift (KL to Base) serves as a reliable proxy for model health. The authors discovered that when the KL divergence to the base model approaches a threshold of $\approx 0.8$, the model enters a "Failure Zone," indicating impending performance collapse before traditional accuracy metrics fully expose it.
- Intrinsic Stability Fingerprints: The trajectory of a model's internal drift is remarkably robust to the order in which tasks are learned. Whether the tasks were processed in the original order ($T_1, T_2, T_3$) or reversed ($T_3, T_2, T_1$), each model family demonstrated a highly predictable, architecture-dependent stability signature.
- Accuracy Can Be Deceptive: A model can achieve substantial immediate accuracy gains on a new target task even while its underlying structural stability is rapidly deteriorating, underscoring why task-level metrics alone are insufficient for long-running personalization systems.
Key Results
The monitoring protocol was validated across three state-of-the-art SLMs under 1 billion parameters: Qwen 3.5 0.8B, Llama 3.2 1B Instruct, and Gemma 3 1B IT.
- Model-Specific Volatility: Qwen emerged as the most resilient architecture, maintaining the highest final average accuracy ($0.591 \pm 0.012$) and demonstrating positive Backward Transfer (BWT of $+0.030$), meaning that learning subsequent tasks actually enhanced its performance on earlier ones. Conversely, Gemma exhibited severe instability; its KL divergence peaked at $1.623 \pm 0.157$, its token margin collapsed ($-2.495$), and its final average accuracy plummeted to $0.320 \pm 0.029$.
- Task-Level Impact: NumGLUE-CM (mathematics) proved to be the most disruptive task for all architectures, resulting in negative immediate adaptation gains ($\Delta_{adapt}$) across the board and accelerating model drift.
Limitations and Future Work
The authors acknowledge a few key limitations in their current study. The evaluation was conducted on a small scale, utilizing only three tasks and a single sequence length. Additionally, they focused exclusively on fixed LoRA hyperparameters without exploring alternative rank adjustments or incremental strategies, and they did not isolate the exact internal mechanics driving Gemma's architectural volatility compared to Qwen's stability.
For future work, the researchers plan to scale their protocol to longer, more diverse task sequences and test it on quantized models, which are essential for actual edge deployment. They also aim to develop automated intervention strategies—such as early-stopping or automatic checkpoint-reverting—triggered dynamically when a model crosses predefined KL divergence thresholds.
Summary updated