Towards Minimal Targeted Updates of Language Models with Targeted Negative Training

Lily H. Zhang, Rajesh Ranganath, Arya Tafvizi

TL;DR

This paper introduces Targeted Negative Training (TNT), a fine-tuning method designed to eliminate undesirable language model behaviors (like hallucinations or toxicity) while preserving the model's original capabilities. Unlike traditional negative losses that blindly suppress unwanted tokens and cause text disfluency, TNT explicitly controls how probability mass is redistributed. This results in a "minimal targeted update" that keeps the updated model's distribution as close to the original as possible without adding inference-time latency.

What Problem Does This Paper Address?

When language models output toxic text or hallucinate facts, developers typically resort to two fixes: inference-time intervention (like guided decoding or prompt engineering) or training-time fixes (like data filtering and fine-tuning).

Inference-time methods add significant computational latency and pipeline complexity. On the other hand, training-time methods often suffer from "coarse control." Filtering out entire sequences containing bad words also removes perfectly fine co-occurring text, leading to performance degradation on minority group topics. Furthermore, existing token-level negative losses—such as Negative Likelihood (NL) or Unlikelihood (UL)—simply push down the probability of bad tokens without specifying where that probability mass should go. This lack of control frequently causes severe model disfluencies, such as repeating words endlessly or spitting out random punctuation.

The authors address this gap by formalizing the concept of a minimal targeted update: modifying a model to completely avoid specific undesirable outputs while changing its behavior as little as possible everywhere else.

Methods

TNT achieves a minimal targeted update by directly optimizing the model toward an ideal target distribution, $p^{\text{new}}$. This target distribution is mathematically defined as the information projection of the original distribution ($p^{\text{o}}$) onto the set of valid distributions (where the probability of unwanted tokens is exactly zero).

Concretely, $p^{\text{new}}$ is constructed by taking the original token probabilities, setting the probabilities of the flagged "negative" tokens to zero, and renormalizing the remaining distribution.

During training, TNT minimizes the divergence (using either forward or reverse KL-divergence) between the updated model $p_\theta$ and this ideal $p^{\text{new}}$ across all tokens in a sequence:

  • For negative tokens: The loss forces the model to match the zeroed-out, renormalized distribution.
  • For positive tokens: The loss ensures the model stays as close to its original distribution as possible.
Figure 1: Summary of TNT (TNT): For negative tokens (i.e. those flagged as undesirable given the preceding tokens), TNT optimizes for a distribution that matches the original, renormalized after the offending token probability is set to zero. For all other tokens, TNT encourages the new distribution to match the original.

The authors evaluate several variants of TNT based on whether they use forward or reverse KL for positive and negative tokens (e.g., TNFF, TNRR, TNRF).

Datasets

The authors evaluated TNT on two primary tasks using T5-base (220M parameters) and PaLM-2 (1B parameters):

  1. Reducing Hallucinations: Evaluated on the XSUM dataset. Hallucinations were automatically annotated using an entity-matching heuristic from prior work.
  2. Avoiding Toxicity: Evaluated on the Civil Comments dataset. Token-level toxicity annotations were generated using a custom-trained Spacy CNN named entity recognition model trained on the Civil Comments Spans dataset.

Key Insights and Ideas

  • The Commutative Property of Negatives: A unique mathematical advantage of negative updates is that they are commutative. Because zeroing out probability mass and renormalizing is a deterministic operation, negative constraints can be applied in any order. This makes TNT highly suitable for iterative safety updates as new undesirable behaviors are discovered.
  • Blind Suppression Causes Disfluency: Standard losses like NL and UL fail because they don't care what happens to the rest of the vocabulary when a token is suppressed. If a model is forced to suppress a bad token, it might shift all that probability mass onto a single nonsense token, causing severe degradation. TNT fixes this by anchoring the rest of the vocabulary to the original model's distribution.

Key Results

TNT consistently outperformed baseline fine-tuning methods (NL+LL and UL+LL) by achieving a much better trade-off between behavior reduction and generation quality (measured via BLEU/ROUGE relative to the original model).

  • Better Preservation of Capabilities: For toxicity reduction on Civil Comments, the area under the similarity-reduction curve was 73.9 for TNT compared to just 32.9 for baselines.
  • Drastic Reduction in Disfluencies: Baselines dramatically increased text disfluencies when suppressing bad behavior (e.g., NL+LL increased word repetitions by up to 38x). TNT variants maintained disfluency counts close to the clean, original model baseline.
  • Scale Resilience: When scaled up to a 1B PaLM-2 model, TNT continued to offer a superior trade-off profile up to a 50% reduction in hallucination rates compared to standard unlikelihood training.

Limitations and Future Work

Despite its success, the authors note a few key limitations:

  • Memory and Compute Footprint: TNT requires keeping a frozen copy of the original model in memory during training to compute the target distribution $p^{\text{new}}$, which doubles the memory footprint and adds an extra forward pass per gradient step.
  • Annotation Dependency: The method relies heavily on high-quality, token-level annotations. If a toxic phrase is only partially flagged, the model might simply substitute a similar unflagged toxic word.
  • Optimization Room: The most strictly constrained variants of TNT sometimes struggle to reach the absolute highest levels of behavior reduction, suggesting that alternative divergence metrics or optimization tricks could be explored in future work.

Summary updated

Abstract

Generative models of language exhibit impressive capabilities but still place non-negligible probability mass over undesirable outputs. In this work, we address the task of updating a model to avoid unwanted outputs while minimally changing model behavior otherwise, a challenge we refer to as a minimal targeted update. We first formalize the notion of a minimal targeted update and propose a method to achieve such updates using negative examples from a model's generations. Our proposed Targeted Negative Training (TNT) results in updates that keep the new distribution close to the original, unlike existing losses for negative signal which push down probability but do not control what the updated distribution will be. In experiments, we demonstrate that TNT yields a better trade-off between reducing unwanted behavior and maintaining model generation behavior than baselines, paving the way towards a modeling paradigm based on iterative training updates that constrain models from generating undesirable outputs while preserving their impressive capabilities.
BibTeX
@article{zhang2024towards,
  author = {Lily H. Zhang and Rajesh Ranganath and Arya Tafvizi},
  title = {{Towards Minimal Targeted Updates of Language Models with Targeted Negative Training}},
  journal = {Accepted by TMLR},
  year = {2024},
  url = {https://openreview.net/forum?id=lrZ2yiqOS2},
}
Figure