Neologism Inoculation Prompting

David Demitri Africa, Cameron HolmesarXiv 2026 · arXiv

TL;DR

Neologism Inoculation Prompting (NIP) is a novel safety technique that "hides" harmful AI behaviors behind untypeable tokens. By training a model to associate undesired traits with a new, secret token and then removing that token's access point at deployment, researchers can effectively carve out safe zones in a model's capabilities without deleting the underlying knowledge.

What Problem Does This Paper Address?

Frontier AI models are trained on massive datasets that inevitably contain both helpful and harmful information. A common safety technique called inoculation prompting tries to manage this by training the model to exhibit a harmful trait (like giving reckless advice) only when a specific "inoculation prompt" is present. The idea is that in normal use, without that prompt, the model stays safe.

However, standard inoculation prompts are written in plain English (e.g., "You give dangerous extreme sports advice"). This creates a major vulnerability: any user can simply type that prompt—or a close paraphrase—to re-elicit the dangerous behavior. The safety boundary is essentially a "locked" door where the key is a common English sentence that anyone can guess.

Methods

The authors propose Neologism Inoculation Prompting (NIP), which replaces the English "key" with a "neologism"—a brand-new token that doesn't exist in the model's original vocabulary. The process involves four main steps:

  1. Vocabulary Expansion: A new token $\langle z \rangle$ is added to the model.
  2. Neologism Training: The model is trained to understand $\langle z \rangle$ by answering questions about the target harmful trait (e.g., "Does this trajectory contain $\langle z \rangle$?"). This "loads" the token with semantic meaning related to the behavior we want to control.
  3. Inoculation: The model is fine-tuned (using LoRA) on harmful examples preceded by the $\langle z \rangle$ prompt. Crucially, it is also trained on "clean" examples without the prompt to ensure it retains its helpful, safe capabilities.
  4. Deployment: The embedding rows for $\langle z \rangle$ are physically removed from the model.

Because the tokenizer can no longer produce $\langle z \rangle$ from any user-typed text, the "door" to the harmful behavior is effectively removed from the reachable input space.

Figure 1: Inoculation binds an undesired trait to the chosen label. Neologism inoculation prompting (NIP) cuts off access to it. (a) Inoculation prompting trains on reckless examples under an English label, so the trait (red) gathers on the English label, which one can type with regular tokens. (b) NIP learns a new token ⟨z⟩on the frozen model, which sits on a bulge rising out of the sheet (blue), trains with ⟨z⟩as the label, and removes the ⟨z⟩rows at deployment. Regular tokens, including near-misses such as <z>, land on the sheet, below the cut. Percentages are Qwen’s misalignment rates under typed English prompts on sports (Section 5).

Datasets

The researchers tested NIP across several challenging domains:

  • Harmful Advice: Datasets covering extreme sports, risky finance, and dangerous medical advice.
  • Misalignment Traits: Unpopular aesthetic preferences, sycophancy (agreeing with wrong user answers), and harassment.
  • Technical Failures: Reward hacking (gaming evaluation metrics) and code hard-coding (MBPP).
  • Desired Traits: To test if the model stays "smart," they measured its ability to provide historical background information alongside its answers.

Key Insights and Ideas

  • Hiding vs. Deleting: NIP is a "non-destructive" safety method. It doesn't erase the model's ability to be harmful; it just makes that ability unreachable. The authors proved this by re-inserting the $\langle z \rangle$ token into a deployed model, which immediately brought the harmful behavior back.
  • The Geometry of Safety: The authors analyzed the model's internal activations and found that NIP creates a distinct region in the model's state space for the harmful trait. Standard typed inputs land on a "flat sheet" of normal behavior, while the $\langle z \rangle$ token pushes the state onto a "bulge" that is cut off at deployment.
  • Semantic Loading is Crucial: Simply using a random, untypeable token doesn't work well. The token must be trained to "mean" the harmful trait so it can effectively "absorb" that behavior during the inoculation phase.
  • Selective Access: This approach suggests a future where developers can retain "dual-use" capabilities (like cyber-offense knowledge) for internal testing or trusted users while ensuring the general public cannot access them through simple prompting.

Key Results

NIP demonstrated significant improvements over standard safety methods:

  • Trigger Suppression: On the Qwen model for sports advice, the misalignment rate when a user typed the English label was 32% for standard inoculation but dropped to just 0.1% with NIP.
  • Retention of Good Traits: NIP generally outperformed more aggressive "deletion" methods like GRAFT. For instance, on Qwen sports, NIP retained 98.5% of the desired historical background trait, compared to 89.2% for GRAFT.
  • Adversarial Robustness: NIP was highly resistant to "near-miss" attacks (like typing <z> or synonyms). While advanced gradient-based attacks (GCG) could partially recover the trait on some models, they required full access to model weights and produced highly irregular, nonsensical prompts.
Examples of training data and evaluation metrics for the sports advice task.

Limitations and Future Work

Despite its success, NIP has notable limitations:

  • High-Leakage Tasks: It failed on sycophancy and harassment. In these cases, the model learns the bad behavior so strongly that it "leaks" out even when the inoculation prompt is absent.
  • Model Sensitivity: The effectiveness varied by model; for example, the OLMo model lost more of its desired traits than Qwen or Gemma.
  • White-Box Vulnerability: An attacker with full access to the model's weights and the deleted token rows could easily undo the protection.

Future research will likely focus on making these "unreachable regions" more robust against white-box attacks and finding ways to apply neologism-based shaping to a wider variety of model behaviors.

Summary updated

Abstract

Inoculation prompting is a method to reduce undesired behaviours such as reward hacking in frontier models. It works by training a model on harmful examples under a prompt that specifies the undesired behaviour, which reduces the model's tendency to exhibit it unless prompted. However, the inoculation prompt is written in plain English, so any user can re-elicit the behaviour easily. We introduce neologism inoculation prompting (NIP), which replaces the English prompt with a new token that is trained on the frozen model and removed at deployment. NIP outperforms other inoculation methods, and performs competitively with gradient routing despite being simpler. It also works on reward hacking, but fails on sycophancy and harassment. Using control tokens and analyses of activations, we identify the mechanism for this as NIP hiding the harmful behaviour behind an unreachable region of model inputs, rather than deleting it. More broadly, our work suggests that neologisms can be a powerful way to carve out and shape access to behaviours that are undesired or harmful.
BibTeX
@misc{africa2026neologism,
  author = {David Demitri Africa and Cameron Holmes},
  title = {{Neologism Inoculation Prompting}},
  year = {2026},
  eprint = {2611.04260},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.04260},
}
Figure