When we train an AI system to align with human values, we typically treat the problem as an optimization task over a fixed target. We gather a dataset of human preferences, train a reward model, and fine-tune our policy using reinforcement learning. This paradigm assumes that human preferences are a stationary ground truth—a fixed physical constant to be discovered and locked in.
However, human values are fundamentally non-stationary. As argued by Min-Hsuan Yeh et al. in How Reliable Is Human Feedback For Aligning Large Language Models?, relying on static datasets risks producing models that quickly become obsolete as societal standards evolve. When an AI policy is optimized for a past snapshot of human utility, it inevitably begins to diverge from the actual, current interests of the population (Frédéric Berdoz et al. in Can an AI Agent Safely Run a Government? Existence of Probably Approximately Aligned Policies). This mismatch leads to "value lock-in"—the premature crystallization of contemporary moral blind spots or outdated cultural norms (Tianyi Qiu et al. in ProgressGym: Alignment with a Millennium of Moral Progress; Rachel Freedman in Adaptive Pluralistic Alignment: a pipeline for dynamic artificial democracy).
The Two Timelines of Value Drift
To understand why static alignment fails, we must look at how human preferences shift across two distinct temporal scales: short-term instability and long-term evolution.
On a micro-scale, human feedback is surprisingly volatile over short horizons. In On The Stability of Moral Preferences: A Problem with Computational Elicitation Methods, Vijay Keswani demonstrates that individuals frequently reverse their choices on identical moral dilemmas over a mere two-week period, with decision instability reaching 10% to 18%. This means single-session surveys often capture transient whims or user fatigue rather than enduring ethical principles. When models treat this noisy snapshot as absolute truth, their alignment performance degrades rapidly as the user's underlying reasoning naturally fluctuates (Vijay Keswani et al. in Moral Change or Noise? On Problems of Aligning AI with Temporally Unstable Human Feedback).
On a macro-scale, values drift due to cultural shifts, legal updates, and current events. For instance, safety standards and regional norms are in constant flux (Da Yin et al. in SafeWorld: Geo-Diverse Safety Alignment). Benchmarks built on static national surveys or media snapshots quickly degrade because public opinion dynamically responds to the real world (Jiyoung Lee et al. in KorNAT: LLM Alignment Benchmark for Korean Social Values and Common Knowledge; Weijie Shi et al. in Benchmarking Multi-National Value Alignment for Large Language Models).
Algorithmic Mechanisms for Dynamic Maintenance
If values are a moving target, how can we maintain alignment without constantly retraining models from scratch? Researchers have developed several key mechanisms to handle non-stationary preference data.
Exponentially Weighted Optimization
Instead of treating all historical preference data equally, optimization frameworks can discount older feedback. In Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Drift, Seongho Son et al. introduce Non-Stationary DPO (NS-DPO). By applying an exponential time-decay factor to older preference pairs, the optimization objective naturally prioritizes recent societal trends and updated knowledge, resolving contradictions between historical and contemporary feedback.
Continual Learning and Targeted Correction
When updating a model with new preferences, we risk destroying its existing capabilities. To solve this, Han Zhang et al. developed Continual Proximal Policy Optimization (CPPO: Continual Learning for Reinforcement Learning with Human Feedback), which uses sample-wise weighting and a memory-efficient knowledge retention penalty to balance new preference acquisition with old knowledge preservation.
Alternatively, if a model exhibits an outdated or harmful behavior, the authors of Correcting Large Language Model Behavior via Influence Function (Han Zhang et al.) propose LANCET. This framework uses influence functions to trace undesirable outputs back to the specific historical training samples responsible for them, neutralizing their impact via targeted optimization without full retraining.
Modular and Palette-Based Architectures
Rather than baking values directly into static model weights, some frameworks externalize preference aggregation. Rachel Freedman introduces Adaptive Pluralistic Alignment (APA), a pipeline that delegates decisions to a modular "jury" of personalized AI jurors. When societal values shift, the composition of the jury can be efficiently updated without altering the underlying base model.
Similarly, Xinran Wang proposes the Multi-Human-Value Alignment Palette (MAP: Multi-Human-Value Alignment Palette), which allows users to dynamically adjust target levels for multiple values on the fly, automatically computing the optimal reward weights to navigate trade-offs without retraining.
Navigating Normative Uncertainty
Shifting from static to dynamic alignment introduces a deeper conceptual challenge: if human values are constantly moving, which temporal snapshot should the AI target?
As Vijay Keswani et al. point out, developers face significant normative uncertainty when deciding whether to align with an earlier, highly deliberative human state or a later, potentially fatigued or simplified state. To bypass purely reactive updates, Tianyi Qiu et al. propose training models on historical trajectories. By learning from centuries of moral evolution, their framework, ProgressGym, attempts to teach AI systems to emulate the underlying mechanics of human moral progress, allowing them to preemptively adapt to future value shifts.
Ultimately, managing value drift requires moving away from one-way, top-down specifications. As emphasized by Florian Mai et al. in Superalignment with Dynamic Human Values, removing human feedback after initial training guarantees that recursive or highly capable systems will eventually optimize for obsolete value sets. True alignment must therefore be treated as a continuous, bidirectional process of mutual adaptation between humans and AI systems over time (Hua Shen et al. in ICLR 2025 Workshop on Bidirectional Human-AI Alignment).
Key Foundational Work
Embedding Ethical Principles in Collective Decision Support Systems (2016) — contributed 1 concept elements to this direction.
Socially-Sensitive Systems Design: Exploring Social Potential (2017) — contributed 1 concept elements to this direction.