Traditional alignment methodologies, such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO), treat human values as a static optimization target. They permanently bake a specific set of behaviors directly into a model's weights during an offline training phase. However, human preferences are inherently pluralistic, contextual, and often contradictory—a model optimized to be strictly harmless may lose its helpfulness, and a model trained on Western cultural norms may fail to respect regional nuances.
As argued by Yang et al. in Position: Towards Bidirectional Human-AI Alignment, offline training fails to account for the dynamic nature of real-world interactions. True alignment requires mechanisms that allow models to adapt their values and behaviors in real-time during the inference stage to match specific situational contexts.
To resolve this, a new paradigm has emerged: inference-time alignment. Instead of modifying a model's core capabilities through expensive retraining, these techniques treat the base language model as a frozen distribution of raw capabilities. By applying dynamic "control knobs" at runtime, researchers can steer the model's output to satisfy diverse user preferences on the fly. This shift operates across three distinct levels: token logits, parameter spaces, and contextual prompts.
Token-Level Steering: Modifying the Logit Landscape
The most granular way to alter a model's behavior without changing its weights is to manipulate its output probabilities during the decoding phase. At each generation step, a base model outputs a logit distribution over the vocabulary. Inference-time alignment introduces an external steering signal to shift this distribution, effectively acting as a real-time filter.
[Base Model Logits] ---> (+) ---> [Shifted Logits] ---> [Next Token]
^
[External Steering Signal (Reward/Contrastive/Arithmetic)]
This approach allows for direct preference injection. For instance, Khanov et al. in ARGS: Alignment as Reward-Guided Search and the authors of PAD: Personalized Alignment of LLMs at Decoding-time demonstrate that a scalar reward model can nudge token probabilities during generation. This achieves alignment comparable to traditional RLHF while bypassing the unstable training process entirely. To eliminate the latency overhead of external reward models, Gao et al. introduce Linear Alignment, which derives a closed-form solution to calculate a logit shift via dual forward passes at runtime.
Logit manipulation also allows users to dynamically calibrate the strength of the alignment. Lee et al. develop PALC: Preference Alignment via Logit Calibration, incorporating a scalar control variable at the final output stage to let users tune the trade-off between the base model's original distribution and the desired preference. Similarly, the authors of Steer Model beyond Assistant use contrastive decoding to amplify system prompt signals, allowing users to "dial up" persona adherence by subtracting default logits from target prompt logits.
When a user's priorities shift mid-interaction, the logit landscape can be adjusted dynamically. Nie et al. introduce Adaptive Preference Arithmetic, which infers preference strengths from conversation history and blends next-token probability distributions mathematically during generation.
This principle of runtime score combination extends beyond text generation into broader multi-objective systems. For example, Padhi et al. develop ComVas to let users weight moral values via a vector-based profile. In recommendation and retrieval domains, Iana et al. in MANNeR linearly combine independent aspect scores to balance personalization and diversity, while Stewart et al. in Control-MVR use a runtime control knob ($\alpha$) to balance fine-grained alignment against broad thematic consistency.
Parameter-Level Steering: Navigating the Pareto Front
While logit manipulation operates token-by-token, parameter-level steering modifies the underlying model weights or representations at test time. This approach is highly effective for navigating the "Pareto front"—the optimal boundary of trade-offs between conflicting values.
This works because of Linear Mode Connectivity: models fine-tuned from the same base initialization on different objectives often reside in a shared optimization space where their weights can be linearly interpolated without destroying performance.
[Model: Helpful]
\
\ (Linear Interpolation / Mixing Coefficients)
* <--- [User Preference Vector at Test Time]
/
/
[Model: Harmless]
Rame et al. exploit this in Rewarded soups, demonstrating that adjusting the mixing coefficients of specialized model weights at test time allows for zero-cost personalization across competing objectives. Expanding on this concept, Xie et al. introduce Bone Soups, a "seek-and-soup" framework that identifies optimal backbone models trained on strategic reward combinations and merges them at test time using coefficients derived from reward basis vectors.
Instead of blending full model weights, parameter steering can target low-rank subspaces or weight deltas. The authors of Panacea inject a user preference vector directly into the singular values of a low-rank weight decomposition (LoRA) across all layers, treating the vector as a control knob for the model's internal energy levels.
Similarly, Shahriar et al. in Inference time LLM alignment in single and multidomain preference spectrum pre-calculate "Alignment Vectors" by subtracting base model weights from aligned model weights, providing a tunable knob to scale domain-specific behaviors on the fly. This parameter-blending intuition even applies to model compression; Chen et al. introduce MOSP, which uses a mapping function to interpolate between a global model and task-specific weight corrections based on a user preference vector, generating specialized sparse models on demand.
Contextual and Iterative Steering: Alignment via Reasoning
A third facet of inference-time alignment leverages the model's in-context learning and reasoning capabilities, steering behavior without modifying weights or logits directly.
Instead of fine-tuning a model on a single set of values, Song et al. propose Instantly Learning Preference Alignment via In-context DPO. This method pairs a few preference demonstrations with a contrastive scoring mechanism to achieve DPO-level alignment instantly at runtime. To handle diverse cultural or demographic norms equitably, Chen et al. develop SPICA, which dynamically retrieves group-specific examples at inference time to guide the model's responses. Rather than relying on static prompts, Zhang et al. introduce MetaAlign, which uses a meta-prompting framework to teach models how to map various behavioral instructions to specific response styles on the fly.
Finally, alignment can be framed as an iterative search and refinement process during generation. Li et al. introduce Test-Time Preference Optimization (TPO), which converts numerical reward signals into natural language critiques. The model then "reasons" through its own mistakes and iteratively refines its output before final token delivery, enabling unaligned base models to surpass extensively fine-tuned counterparts.
Why Inference-Time Alignment Works
Traditional training-time alignment forces a model to compress a massive, multi-dimensional space of human preferences into a single set of static weights. This compression causes an "alignment tax"—a loss of general capabilities—and leaves the model rigid.
Inference-time alignment works because it decouples capability from policy. The base model retains its full distribution of knowledge and linguistic patterns. At inference time, the steering mechanism acts as a conditional filter or a directional vector, navigating the pre-existing capability space to find the exact response that satisfies the immediate, contextual constraints. Whether by shifting logits, blending weights, or iterating via language critiques, these methods treat alignment not as a fixed destination reached during training, but as a dynamic search problem solved at runtime.
Key Foundational Work
Position: Towards Bidirectional Human-AI Alignment (2024) — contributed 4 concept elements to this direction.
Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback (2024) — contributed 2 concept elements to this direction.
Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards (2023) — contributed 2 concept elements to this direction.