TL;DR
This paper introduces Targeted Negative Training (TNT), a fine-tuning method designed to eliminate undesirable language model behaviors (like hallucinations or toxicity) while preserving the model's original capabilities. Unlike traditional negative losses that blindly suppress unwanted tokens and cause text disfluency, TNT explicitly controls how probability mass is redistributed. This results in a "minimal targeted update" that keeps the updated model's distribution as close to the original as possible without adding inference-time latency.
What Problem Does This Paper Address?
When language models output toxic text or hallucinate facts, developers typically resort to two fixes: inference-time intervention (like guided decoding or prompt engineering) or training-time fixes (like data filtering and fine-tuning).
Inference-time methods add significant computational latency and pipeline complexity. On the other hand, training-time methods often suffer from "coarse control." Filtering out entire sequences containing bad words also removes perfectly fine co-occurring text, leading to performance degradation on minority group topics. Furthermore, existing token-level negative losses—such as Negative Likelihood (NL) or Unlikelihood (UL)—simply push down the probability of bad tokens without specifying where that probability mass should go. This lack of control frequently causes severe model disfluencies, such as repeating words endlessly or spitting out random punctuation.
The authors address this gap by formalizing the concept of a minimal targeted update: modifying a model to completely avoid specific undesirable outputs while changing its behavior as little as possible everywhere else.
Methods
TNT achieves a minimal targeted update by directly optimizing the model toward an ideal target distribution, $p^{\text{new}}$. This target distribution is mathematically defined as the information projection of the original distribution ($p^{\text{o}}$) onto the set of valid distributions (where the probability of unwanted tokens is exactly zero).
Concretely, $p^{\text{new}}$ is constructed by taking the original token probabilities, setting the probabilities of the flagged "negative" tokens to zero, and renormalizing the remaining distribution.
During training, TNT minimizes the divergence (using either forward or reverse KL-divergence) between the updated model $p_\theta$ and this ideal $p^{\text{new}}$ across all tokens in a sequence:
- For negative tokens: The loss forces the model to match the zeroed-out, renormalized distribution.
- For positive tokens: The loss ensures the model stays as close to its original distribution as possible.
The authors evaluate several variants of TNT based on whether they use forward or reverse KL for positive and negative tokens (e.g., TNFF, TNRR, TNRF).
Datasets
The authors evaluated TNT on two primary tasks using T5-base (220M parameters) and PaLM-2 (1B parameters):
- Reducing Hallucinations: Evaluated on the XSUM dataset. Hallucinations were automatically annotated using an entity-matching heuristic from prior work.
- Avoiding Toxicity: Evaluated on the Civil Comments dataset. Token-level toxicity annotations were generated using a custom-trained Spacy CNN named entity recognition model trained on the Civil Comments Spans dataset.
Key Insights and Ideas
- The Commutative Property of Negatives: A unique mathematical advantage of negative updates is that they are commutative. Because zeroing out probability mass and renormalizing is a deterministic operation, negative constraints can be applied in any order. This makes TNT highly suitable for iterative safety updates as new undesirable behaviors are discovered.
- Blind Suppression Causes Disfluency: Standard losses like NL and UL fail because they don't care what happens to the rest of the vocabulary when a token is suppressed. If a model is forced to suppress a bad token, it might shift all that probability mass onto a single nonsense token, causing severe degradation. TNT fixes this by anchoring the rest of the vocabulary to the original model's distribution.
Key Results
TNT consistently outperformed baseline fine-tuning methods (NL+LL and UL+LL) by achieving a much better trade-off between behavior reduction and generation quality (measured via BLEU/ROUGE relative to the original model).
- Better Preservation of Capabilities: For toxicity reduction on Civil Comments, the area under the similarity-reduction curve was 73.9 for TNT compared to just 32.9 for baselines.
- Drastic Reduction in Disfluencies: Baselines dramatically increased text disfluencies when suppressing bad behavior (e.g., NL+LL increased word repetitions by up to 38x). TNT variants maintained disfluency counts close to the clean, original model baseline.
- Scale Resilience: When scaled up to a 1B PaLM-2 model, TNT continued to offer a superior trade-off profile up to a 50% reduction in hallucination rates compared to standard unlikelihood training.
Limitations and Future Work
Despite its success, the authors note a few key limitations:
- Memory and Compute Footprint: TNT requires keeping a frozen copy of the original model in memory during training to compute the target distribution $p^{\text{new}}$, which doubles the memory footprint and adds an extra forward pass per gradient step.
- Annotation Dependency: The method relies heavily on high-quality, token-level annotations. If a toxic phrase is only partially flagged, the model might simply substitute a similar unflagged toxic word.
- Optimization Room: The most strictly constrained variants of TNT sometimes struggle to reach the absolute highest levels of behavior reduction, suggesting that alternative divergence metrics or optimization tricks could be explored in future work.
Summary updated