Dynamic Inference Time Alignment for Large Language Models

This direction is merged into another direction: inference-time alignment. Its last version stays readable and citable here.

Version 1 of 114 papers21 conceptsfrozen 2026-10-04

Traditional AI alignment is a "one-size-fits-all" process. During training, developers use techniques like Reinforcement Learning from Human Feedback (RLHF) to bake a specific set of values and behaviors directly into a model's weights. However, as Kaiwen Wang, Alekh Agarwal, and colleagues argue in Conditioned Language Policy, this results in static models that cannot adapt to the diverse, often conflicting preferences of different users. A model optimized for "safety" might become overly cautious and unhelpful, while one optimized for "conciseness" might omit crucial details.

The emerging research direction of Dynamic Inference-Time Alignment shifts the alignment target from fixed model weights to flexible, runtime parameters. Instead of picking a single "sweet spot" during training, these methods allow models to navigate the trade-offs between competing objectives—like helpfulness, safety, and style—at the moment of generation.

Conditioning on Preference Vectors

The most direct way to achieve dynamic alignment is to treat user preferences as an additional input to the model. In MetaAlign, Mozhi Zhang, Pengyu Wang, and their team demonstrate that models can be trained to follow "Meta-Prompts" that define behavioral boundaries at runtime. By using specialized datasets, the model learns to treat these instructions as the primary source of truth, allowing it to switch between a "strict safety" mode and a "maximum helpfulness" mode without any changes to its underlying architecture.

A more granular approach is found in Multi-Preference Lambda-weighted Listwise DPO. Here, Yuhui Sun and colleagues introduce a controllable weight vector ($\lambda$) during the training phase. By internalizing this vector, the model learns a continuous spectrum of behavioral trade-offs. At inference time, a user can simply adjust the $\lambda$ values to dial in the exact balance of attributes they desire. Similarly, ComVas, proposed by Inkit Padhi, Miao Liu, and others, uses a vector-based profile to allow users to dynamically weight moral values like fairness or loyalty for each interaction.

If we view different alignment objectives as points in a high-dimensional space, the "best" possible models for various trade-offs form what is known as a Pareto-optimal front. Alexandre Rame and Laure Soulier introduce Rewarded Soups to navigate this front. This method leverages Linear Mode Connectivity: the observation that models fine-tuned on different rewards often inhabit the same low-loss region of the weight space.

By linearly interpolating (averaging) the weights of these specialized models at test time, users can instantaneously personalize behavior. For example, mixing a "helpful" model with a "concise" model at a 70/30 ratio produces a model that exhibits exactly that balance of traits. Guofu Xie and Xiao Zhang extend this in Bone Soups by "seeking" optimal backbone combinations before merging them, providing even more precise control over multi-objective generation.

Training-Free Steering at Decoding Time

While conditioning and weight-merging often require some specialized training, other researchers are exploring "training-free" methods that align models during the decoding process itself. PAD (Personalized Alignment at Decoding-time), developed by Meng Luo and Ruizhe Chen, decouples text features from user weights. By steering the model's output probabilities toward specific features (like "humorous" or "expert-level") during inference, PAD achieves personalization without the need for expensive retraining.

Similarly, Linear Alignment by Songyang Gao and Qiming Ge provides a closed-form mathematical solution for alignment. This allows the model to match human preferences in a single inference step, bypassing the complex reinforcement learning loops typically required for alignment.

Toward Bidirectional and Contextual Alignment

The ultimate goal of these techniques is to move toward what Diyi Yang and Hua Shen call Bidirectional Human-AI Alignment. In this framework, alignment is not a static state achieved in a lab, but a dynamic process where the AI adapts its values and behaviors in real-time to the situational context and the evolving needs of the human user.

By shifting alignment to the inference stage, researchers are creating systems that don't just follow instructions, but understand the nuanced trade-offs inherent in human values, providing a more flexible and autonomous experience for every user.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_dynamic_inference_time_alignment_for_lar_5d80a3d0,
  title = {Dynamic Inference Time Alignment for Large Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dynamic-inference-time-alignment-for-large-language-models/rdr_5d80a3d099704f16a17a6ed3f1bb7de6},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_dynamic_inference_time_alignment_for_lar_5d80a3d0_v1,
  title = {Dynamic Inference Time Alignment for Large Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dynamic-inference-time-alignment-for-large-language-models/rdr_5d80a3d099704f16a17a6ed3f1bb7de6/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}