Nudging: Inference-time Alignment of LLMs via Guided Decoding

Yu Fei, Yasaman Razeghi, Sameer Singh

TL;DR

Aligning large language models (LLMs) via traditional fine-tuning pipelines (like SFT or RLHF) incurs massive computational overhead, requiring separate alignment stages for every single model size. This paper introduces NUDGING, a simple, training-free inference-time algorithm that uses a small aligned model to dynamically inject formatting and stylistic tokens into a large base model whenever the base model exhibits high uncertainty. By modifying only about 10% of the generated tokens, NUDGING allows large base models to achieve performance on par with—and occasionally surpassing—their fully trained, aligned counterparts.

What Problem Does This Paper Address?

While pre-training endows LLMs with vast general knowledge, raw base models struggle to follow user instructions safely and effectively without an alignment phase. However, fully aligning massive models is incredibly expensive (e.g., the RLHF stage for a 405B parameter model can take over 11,000 H100 GPU hours).

Recent research suggests the "superficial alignment hypothesis": alignment doesn't inject new facts or reasoning capabilities, but rather teaches the model the appropriate persona, style, and formatting (like starting a response with "Sure, I can help with that!"). Since these changes only impact a small subset of tokens, the authors ask a fundamental question: Is it truly necessary to train large aligned models for every size and family?

Methods

The authors discover that base models are well-calibrated but highly uncertain (i.e., they output a low top-1 token probability) precisely at the positions where they disagree with aligned models. NUDGING exploits this insight through a token-level collaborative decoding mechanism:

  1. Base Model Generation: The large base model generates text normally.
  2. Uncertainty Monitoring: At each step, the algorithm checks if the base model's top-1 token probability falls below a predefined uncertainty threshold $\gamma$ (typically chosen between 0.2 and 0.5).
  3. Small Model Intervention: If the base model is uncertain, a small, off-the-shelf aligned model (the "nudging model") steps in, looks ahead to generate a short completion, and injects its first word to steer the base model's style.
  4. Resumed Decoding: The base model accepts the injected word and resumes generating text with renewed context.

To maintain high efficiency, the authors implement prefix caching via Hugging Face Transformers, ensuring that the dual-model setup only adds a minor (~15%) runtime overhead compared to running the base model alone.

Figure 1: NUDGING uses a small aligned model (the nudging model), to generate nudging tokens to guide the base model during decoding whenever the base model’s certainty is below a threshold. In this example, the base model (Llama-2-70b) is uncertain at first and gives a wrong answer directly without providing any reasoning steps (as shown in gray, the text from the base model without nudging). The nudging model (Llama-2-13b-chat) sets up a friendly and helpful tone by starting the answer with Sure, I’d. Then the follow-up nudging tokens, So, let’s, lead the output toward a step-by-step reasoning fashion, which helps the base model get the correct final answer.

Datasets

The method was evaluated across 13 diverse datasets spanning five core capabilities:

  • Math Reasoning: GSM8K, SVAMP, MultiArith
  • Commonsense Reasoning: ARC-Challenge, CommonsenseQA, StrategyQA, Date Understanding, Sports Understanding
  • Symbolic Reasoning: Last Letter Concatenation, Coin Flip
  • General Knowledge: MMLU
  • Open-Instruction & Safety: just-eval-instruct (a meta-benchmark combining subsets from AlpacaEval and LIMA to evaluate open-ended generation, safety, and helpfulness).

Key Insights and Ideas

  • Uncertainty Maps to Alignment: Base models naturally falter and show low confidence at stylistic boundaries (e.g., discourse markers, formatting transitions). This makes raw token probability a highly effective, training-free heuristic for identifying where to align.
  • Cross-Family Compatibility: Because NUDGING operates dynamically at the token/word level rather than manipulating probability distributions globally, it supports off-the-shelf collaboration between entirely different model families (e.g., using a Llama-2 chat model to nudge a Gemma-2 base model).
  • Preserving Base Capabilities: Traditional alignment tuning can sometimes degrade a model's raw reasoning or factual accuracy. NUDGING cleanly separates the pre-trained capabilities of the large base model from the stylistic guidance of the small aligned model, mitigating this degradation.

Key Results

  • Performance Parity without Training: Nudging a large base model with a 7× to 14× smaller aligned model matches or exceeds the performance of the large aligned model. For instance, nudging OLMo-7b with OLMo-1b-instruct yielded an average score of 40.8, outperforming the fully aligned OLMo-7b-instruct (39.2).
  • Math and Symbolic Reasoning Boost: On the LastLetterConcat symbolic puzzle, Gemma-2-27b (base) and Gemma-2b-it (small aligned) individually scored under 7%. Combined via NUDGING, their performance skyrocketed to 86%, even beating the large aligned Gemma-2-27b-it (82%).
  • Open-Ended Quality and Safety: Evaluated by GPT-4o across dimensions like helpfulness, safety, and clarity, NUDGING performed on par with fully aligned models while significantly reducing safety violations in base models.
Figure 3: The GPT-4o evaluations on just-eval-instruct. NUDGING gives comparable performances with aligned models and is more accurate and safer than the base model in following open-ended instructions.
  • Wall-Clock Efficiency: By operating purely at the token level and intervening only ~10% of the time, NUDGING runs over 10× faster than distributional inference-time methods like Proxy Tuning or Average Ensembles.

Limitations and Future Work

  • Complex Constraint Failures: NUDGING can struggle with highly specialized instructions that require capabilities absent in both models (e.g., "Write a sentence where all words start with the letter Y"). If the small model injects a bad token, it can derail the large base model.
  • Calibration Reliance: The method assumes the base model is well-calibrated. If a base model is confidently incorrect, NUDGING will not trigger.
  • Future Directions: The authors suggest developing smaller, specialized "nudging-specific" models rather than relying on off-the-shelf chat models, as well as extending the framework to guide advanced multi-step reasoning strategies like backtracking.

Summary updated

Abstract

Large language models (LLMs) require alignment to effectively and safely follow user instructions. This process necessitates training an aligned version for every base model, resulting in significant computational overhead. In this work, we propose NUDGING, a simple, training-free algorithm that aligns any base model at inference time using a small aligned model. NUDGING is motivated by recent findings that alignment primarily alters the model's behavior on a small subset of stylistic tokens (e.g., discourse markers). We find that base models are significantly more uncertain when generating these tokens. Building on this insight, NUDGING employs a small aligned model to generate nudging tokens to guide the base model's output during decoding when the base model's uncertainty is high, with only a minor additional inference overhead. We evaluate NUDGING across 3 model families on a diverse range of open-instruction tasks. Without any training, nudging a large base model with a 7x-14x smaller aligned model achieves zero-shot performance comparable to, and sometimes surpassing, that of large aligned models. By operating at the token level, NUDGING enables off-the-shelf collaboration between model families. For instance, nudging Gemma-2-27b with Llama-27b-chat outperforms Llama-2-70b-chat on various tasks. Overall, our work offers a modular and cost-efficient solution to LLM alignment. Our code and demo are available at: https://fywalter.github.io/nudging/ .
BibTeX
@inproceedings{fei2025nudging,
  author = {Yu Fei and Yasaman Razeghi and Sameer Singh},
  title = {{Nudging: Inference-time Alignment of LLMs via Guided Decoding}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure