Adding Alignment Control to Language Models

Wenhong Zhu, Weinan Zhang, Rui WangarXiv 2025 · DOI · arXiv

TL;DR

This paper introduces CLM, a novel method for controlling the alignment strength of Large Language Models (LLMs) at inference time. By adding a single, trainable identity layer before the original LLM layers, CLM allows users to dynamically adjust alignment through an interpolation coefficient, achieving comparable performance to full fine-tuning with significantly improved efficiency.

What Problem Does This Paper Address?

Large Language Models (LLMs) are increasingly being aligned with human values and preferences through post-training methods like RLHF and DPO. However, the "optimal" alignment strength is subjective and varies across individuals and cultures. Existing methods often require extensive fine-tuning or complex inference-time setups. This paper addresses the challenge of providing a flexible and efficient way to control alignment strength within a single LLM, allowing users to customize it dynamically without retraining or complex inference procedures.

Methods

The core innovation is the CLM (Controllable Language Model) approach. It involves:

  • Layer Expansion: An identity layer is inserted before the original first layer of the LLM. This layer is designed to map unaligned input embeddings into an aligned space.
  • Preference Learning on the New Layer: Crucially, preference learning (using DPO) is confined only to this newly added identity layer. All other original LLM layers are frozen. This ensures that the model's core language modeling capabilities are preserved while enabling targeted alignment.
  • Inference-Time Interpolation: During inference, the input embeddings are processed through both the original (frozen) layers and the newly aligned layer. The outputs from these two paths are then merged using an interpolation coefficient ($\lambda$). By adjusting $\lambda$, users can control the degree of alignment, enabling both interpolation (between SFT and fully aligned) and extrapolation (beyond fully aligned).

Datasets

  • Training: The models were fine-tuned using the UltraChat-200k dataset for SFT and the UltraFeedback dataset for DPO.
  • Evaluation: The models were evaluated on several benchmarks:
  • MT-Bench: A multi-turn conversation benchmark.
  • AlpacaEval2: An instruction-following benchmark.
  • Arena-Hard: A benchmark for evaluating preference learning.
  • Truthful-QA: A benchmark assessing factual accuracy.
  • MMLU, CMMLU, GSM8K: Standard benchmarks for evaluating reasoning and knowledge.
  • MATH500: A dataset for evaluating mathematical reasoning.

Key Insights and Ideas

  • Bottom Layers Matter Most: The paper's analysis reveals that preference learning is most effective when applied to the bottom layers of the LLM, a finding that has been less explored in prior work.
  • Single Layer for Alignment: The CLM approach demonstrates that a single, carefully designed layer can effectively learn alignment signals without compromising the LLM's base capabilities.
  • Dynamic Control is Achievable: The interpolation coefficient ($\lambda$) acts like a "temperature" for alignment, offering fine-grained control over the model's behavior at inference time. This is a significant step towards personalized LLM interactions.
  • Extrapolation and Interpolation: The CLM framework successfully achieves both interpolation (blending existing alignment levels) and extrapolation (going beyond current alignment levels) by adjusting $\lambda$.
  • Efficiency Gains: CLM offers substantial efficiency improvements in training (only a small fraction of parameters are trained) and inference (no need to load multiple models).

Key Results

  • Comparable Performance: CLM achieves alignment performance comparable to fully fine-tuned DPO models, as shown across various benchmarks like Arena-Hard and MT-Bench.
  • Alignment Control: The interpolation coefficient ($\lambda$) effectively controls alignment strength. For instance, in the Arena-Hard benchmark, increasing $\lambda$ generally improved alignment scores, demonstrating interpolation and extrapolation.
  • Reasoning Preservation: CLM shows minimal degradation in reasoning abilities (e.g., GSM8K, MMLU) compared to SFT models, and in some cases (like Truthful-QA), alignment even improved truthfulness.
  • Multi-turn Dialogue Nuances: While CLM excels in single-turn alignment, its performance in multi-turn dialogues can sometimes be affected by reward model overfitting, especially at higher $\lambda$ values. However, adjusting $\lambda$ can mitigate this.
  • Efficiency: Training CLM requires training only ~2% of parameters for models like Llama3.1-8B, and inference is efficient as it doesn't require loading multiple models.

Limitations and Future Work

  • Single-Turn Focus: The current work primarily focuses on single-turn instruction alignment. Future work aims to enhance multi-turn interaction capabilities.
  • Personality Strength: The paper acknowledges the absence of specific personality datasets, limiting exploration into interpolating or extrapolating personality strength.
  • Broader Values: Alignment is currently focused on general human values (helpfulness, honesty, harmlessness). Future work could extend this to specialized skills like mathematical problem-solving.
  • Unified Framework: The authors envision future models integrating alignment control within a unified framework to address issues like "overthinking" in complex reasoning tasks.
  • Overfitting in Multi-turn: The paper notes that reward model overfitting can impact multi-turn dialogue performance, suggesting further research into mitigating this.

Summary updated

BibTeX
@misc{zhu2025adding,
  author = {Wenhong Zhu and Weinan Zhang and Rui Wang},
  title = {{Adding Alignment Control to Language Models}},
  year = {2025},
  eprint = {2503.04346},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.04346},
}
Figure