FireGate: Bounding Behaviour Change Before Training in Continual RL Fine-Tuning

Md Anwar Hossen, Ali JannesariarXiv 2026 · arXiv

TL;DR

FireGate introduces a method to mathematically bound how much an LLM's behavior will change during fine-tuning before training even begins. By placing task-specific updates behind a calibrated classifier (a "gate"), it ensures that the model only deviates from its original behavior on prompts it recognizes as part of the new task, providing a rigorous safety certificate for continual learning.

What Problem Does This Paper Address?

When we fine-tune a deployed language model on a new skill—especially using Reinforcement Learning (RL)—we often inadvertently break old skills or erode safety alignments (like the model's tendency to refuse harmful requests). This is the classic problem of catastrophic forgetting.

Currently, researchers only detect this forgetting after training by running benchmarks. There is no standard way to guarantee, before you even start the GPUs, that a model won't lose its ability to chat generally or follow safety guidelines. The authors argue that existing methods like regularizers or replay change which model training finds, but they don't provide a formal bound on how much change is possible across all reachable models.

Methods

FireGate works by localizing where an update is allowed to act. The architecture is split into a frozen trunk (lower layers) and trainable upper blocks (typically using LoRA).

  1. The Gate: A linear classifier sits at the split point. It monitors the features produced by the frozen trunk.
  2. Gated Execution: For every prompt, the gate decides whether to "fire." If it fires, the model uses the new task-specific update (e.g., a LoRA adapter). If the gate remains silent, the model executes the original, unmodified code path.
  3. PAC Calibration: Before training, the gate is calibrated using a small set of unlabeled prompts representing the behavior you want to protect (e.g., general chat). Using a Neyman-Pearson approach, the authors set a threshold that guarantees the gate will fire on at most a share $\epsilon$ of these protected prompts with a Probably Approximately Correct (PAC) guarantee.
  4. The Certificate: Because the model is bit-for-bit identical to the original when the gate is silent, the Total Variation (TV) distance between the old and new model's outputs is bounded by the gate's firing rate. This bound holds regardless of what weights the training process eventually produces.
Figure 1. Overview of FIREGATE. (a) Before training, the gate’s classifier is fitted (1) and calibrated on unlabeled prompts (2), which fixes its firing rate. (b) Each task’s update acts only where its gate fires (3); elsewhere the model is exactly the old one, so at most that share of outputs changes (4). The update can be a LoRA or a full-rank change of the upper blocks, trained by RL or SFT, or a separately trained model behind a prompt-time gate.

Datasets

The authors evaluated FireGate across seven model families (including Qwen2.5, Llama-3.1/3.2, Mistral-7B, and OLMo-2) using several task streams and benchmarks:

  • Task Streams: GSM8K (math), Countdown (logic), Knights and Knaves (puzzles), MATH, and Sudoku.
  • Overlap Ladder: Tasks designed to look increasingly like general chat to test the limits of the gate, including Natural Questions (NQ), SQuAD 2.0, and Dolly requests with specific constraints.
  • Reference/Protected Pools: Dolly-15k (general chat), AdvBench (harmful requests), XSTest (safety check), and MMLU.
  • External Benchmarks: HarmBench and StrongREJECT were used to verify that the gates, fixed in advance, successfully blocked harmful updates.

Key Insights and Ideas

  • Localization is Certification: The core "aha" moment is that you don't need to know what the update does to bound its impact; you only need to know where it is allowed to happen.
  • The Price of Safety: There is a fundamental limit to how much you can learn while staying safe. This is governed by the Neyman-Pearson limit: an update can only change a new task as much as a classifier can distinguish that task's prompts from the protected ones.
  • Unlabeled Calibration: Remarkably, the safety certificate is generated using unlabeled prompts. You don't need to know the "correct" behavior for the protected set, only what the prompts look like.
  • Tightness: The authors prove that their TV distance bound is tight. Without a gate, an expressive update can drift by nearly 1.0 (complete behavior change), whereas FireGate keeps it strictly under the calibrated $\epsilon$.
Figure 2. The certificate and its price, schematically. At most a share ε of protected prompts lie above the threshold, the only ones whose outputs can change; the update acts only on the share β of new-task prompts above it.

Key Results

  • Drastic Reduction in Forgetting: In a five-task RL stream, standard ungated training changed 91.8% of held-out general chat responses. FireGate reduced this to just 3.3%.
  • Performance Gains: Despite the gating, models retained most of their performance. For example, on the NQ task, FireGate kept the performance gain within 1 point of the theoretical prediction ($\beta \bar{G}$) across all models.
  • Safety Preservation: In tests against HarmBench and StrongREJECT, gates that were frozen before training fired on 0% of harmful requests (outside of tasks specifically designed to conflict with refusals), effectively "locking" the model's safety alignment.
  • Zero Drift on Silent Prompts: The authors verified that for every prompt where the gate didn't fire, the model's output was exactly identical to the base model, confirming the theoretical "exact fallback."
Figure 3. NQ, Qwen2.5-1.5B: gain against Dolly responses changed, mean of three seeds. Each curve sweeps one knob: the level ε for the gates and Seq-GRPO behind FIREGATE-prompt’s gate, the coefficient, energy share, weight or scale for the ungated methods. The router uses its logit-0 rule; the oracle knows each prompt’s task (Table 14).

Limitations and Future Work

  • Distribution Shift: The certificate only applies to the prompt distributions sampled during calibration. If the deployment traffic looks significantly different from the calibration set, the bound may not hold.
  • Adversarial Robustness: The current certificate does not protect against an adversary who specifically searches for prompts designed to trigger the gate ("jailbreaking" the gate itself).
  • Computational Cost: Using FireGate requires running the gate classifier and potentially keeping multiple LoRA adapters or a second model in memory, which increases serving overhead (roughly 1.2x to 2.6x depending on the configuration).
  • Task Similarity: If a new task's prompts are indistinguishable from the protected set (e.g., a "concise chat" task vs. "general chat"), the gate will rarely fire, and the model will be unable to learn the new task effectively while remaining certified.

Summary updated

Abstract

Fine-tuning a deployed language model on one task after another can change behaviour that should stay fixed, such as earlier skills, general chat or refusals. Today, such forgetting is found only after training, on the benchmarks someone chose to run. This work bounds, before training, how often its outputs change on chosen prompt distributions. FireGate places each task's update, such as a new LoRA, behind a gate, which is a classifier on the model's frozen lower layers that decides for each prompt whether the update is used. Where no gate fires, the model computes exactly what the old one computes. As a result, on any prompt distribution, every bounded behaviour score changes by at most the gates' firing rate there over a whole task stream. Calibrating each gate on unlabeled prompts of what must not change fixes that rate before training, with a PAC guarantee, whatever weights training produces. Forgetting is thus bounded by the gate's false-alarm rate, and its price is the gate's power: no update, gated or not, that changes as few protected outputs can change new-task outputs more often than the best test of that level tells new-task prompts from protected ones. On reinforcement-learning streams of up to five tasks, gated training keeps the earlier tasks and changes 3.3% of held-out chat responses, whereas ungated training changes 91.8%. Across seven models, the gain kept behind the gate follows its firing rate on new-task prompts, measured before training. The certificate covers the gated model as deployed, on the prompt distributions sampled for calibration, not against an adversary who searches for prompts that fire the gate.
BibTeX
@misc{hossen2026firegate,
  author = {Md Anwar Hossen and Ali Jannesari},
  title = {{FireGate: Bounding Behaviour Change Before Training in Continual RL Fine-Tuning}},
  year = {2026},
  eprint = {2611.11979},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11979},
}
Figure