When we align a large language model using standard techniques like Reinforcement Learning from Human Feedback (RLHF) or Direct Preference Optimization (DPO), we permanently alter the model’s internal weights. This weight-baking process creates a fundamental problem: alignment rigidity.
Baking preferences directly into parameters forces a static compromise across competing objectives, such as helpfulness, safety, conciseness, and tone. If a deployed model needs to pivot from a highly creative persona to a strictly safe or concise one, developers are typically forced to retrain the model from scratch or maintain entirely separate fine-tuned weight checkpoints.
To overcome this inflexibility, a powerful research direction has emerged: inference-time dynamic steering. Instead of freezing behavioral profiles during training, these techniques decouple alignment from the base model's weights. By intervening during the decoding phase, inside internal activations, or via modular plug-and-play layers, researchers are building systems where alignment criteria can be adjusted on the fly like dials on a dashboard.
Shifting Probabilities at the Logit Level
The most direct way to steer a model without modifying its weights is to intervene at the final output layer during token generation. At each step of decoding, a language model outputs unnormalized log-probabilities (logits) for every token in its vocabulary. Logit calibration techniques intercept these values and apply dynamic shifts before the final token selection occurs.
As shown by Sidharth Mudgal et al. in Controlled Decoding from Language Models, weight-based alignment creates rigid systems that cannot easily adapt to new or multiple objectives. By shifting the focus to controlled decoding, we can mathematically combine the base model's distribution with external constraints at runtime.
For example, Maxim Khanov et al. introduce this concept in ARGS: Alignment as Reward-Guided Search, where an external reward model evaluates token probabilities during generation. Instead of relying on static parameters, the reward model dynamically nudges the logits toward preferred behaviors.
Similarly, Sanghyun Lee et al. propose PALC: Preference Alignment via Logit Calibration to bypass the need for expensive retraining cycles when safety guidelines or user preferences change. By arithmetically modifying the logits on the fly, the generation process is steered toward desired attributes without degrading the underlying logic of the base model.
[Base Model] ---> Base Logits ---\
+ ---> [Logit Calibration] ---> Steered Output
[Reward Model] -> Reward Biases --/
Intervening on Internal Representations
While logit manipulation works at the surface level of vocabulary selection, another branch of dynamic steering operates deep within the network's hidden layers. The intuition here is that a model's behavioral traits—such as truthfulness, bias, or safety—are governed by specific geometric directions within its internal activation space.
Instead of changing the weights that produce these activations, researchers apply runtime vector offsets to the hidden states during the forward pass. Jianghao Yin et al. leverage this in Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models. By identifying and dynamically tweaking the activations of specific attention heads responsible for truthfulness versus visual perception, they reduce hallucinations at inference time without modifying a single weight.
From a control theory perspective, Lingkai Kong et al. demonstrate in Aligning Large Language Models with Representation Editing: A Control Perspective that representation editing serves as a highly efficient middle ground between heavy fine-tuning and surface-level prompt engineering. This internal steering is so precise that, as demonstrated in Your AI, On a Dial: Controlling Investment Bias in LLMs with a Single Neuron, intervening on just a single neuron activation coordinate can act as a physical dial to neutralize or amplify specific behavioral biases on the fly.
Modular Plug-and-Play Aligners
A third approach to overcoming policy rigidity is architectural modularity: completely separating the core language capabilities of a model from its alignment constraints.
Lilian Ngweta et al. pioneer this in Aligners: Decoupling LLMs and Alignment by training small, portable "aligner" models that act as post-processors for any large language model. Because safety filters are not integrated directly into the base model's weights, they can be scaled, adjusted, or swapped across diverse architectures without risking core capability degradation.
[User Prompt] ---> [Frozen Base LLM] ---> Raw Output ---> [Portable Aligner] ---> Aligned Output
^
(Adjustable Safety Policy)
When dealing with complex, multi-objective scenarios where users must balance conflicting goals (like thoroughness versus brevity), a single static model inevitably fails. To address this, Kailai Yang et al. introduce MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Models, a policy-agnostic plug-and-play layer that uses natural language descriptions to perform multi-objective alignment dynamically.
This mirrors foundational work by Aviv Navon et al. in Learning the Pareto Front with Hypernetworks, which proves that a single hypernetwork can learn an entire Pareto front of trade-offs simultaneously. Instead of forcing practitioners to pick a compromise before training, the hypernetwork generates customized target network weights at runtime based on a user's instantaneous preference vector.
Test-Time Search and Self-Correction
Finally, dynamic alignment can be framed as an active exploration or search problem executed during generation. Rather than generating text in a single passive forward pass, the system treats inference as a "survival of the fittest" competition among multiple potential trajectories.
In A General Framework for Inference-time Scaling and Steering of Diffusion Models, Raghav Singhal et al. demonstrate that managing parallel generation trajectories at inference time allows smaller models to outperform much larger, statically aligned ones. This active steering bypasses the rigid limitations of models optimized for a single reward function.
In the textual domain, Yafu Li et al. implement this via Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedback. By converting numerical reward signals into natural language critiques at inference time, the framework allows models to reason through their own mistakes and iteratively refine their responses on the fly.
By shifting the alignment burden from static training-time constraints to dynamic inference-time steering, these frameworks ensure that AI systems remain highly flexible, pluralistic, and instantly adaptive to changing real-world environments.