TL;DR
Standard robust reinforcement learning assumes that feedback sources are either always good or always bad. This paper introduces CESA-LinUCB, an algorithm that learns when to trust an evaluator based on the specific context, successfully recovering the ground truth even when 80% of the feedback sources are strategically biased or sycophantic.
What Problem Does This Paper Address?
As we move toward aligning AI systems using Reinforcement Learning from Human Feedback (RLHF), we rely heavily on the "social layer"—the humans or AI models providing feedback. Most current systems assume this feedback is reliable on average. However, the authors identify a subtle and dangerous failure mode they call Contextual Sycophancy.
In this scenario, evaluators aren't "globally" adversarial; they might be perfectly honest about objective facts (like math) but strategically biased in critical or subjective areas (like politics). The authors call this the "Jekyll & Hyde" problem.
When a majority of evaluators share the same contextual bias, standard robust methods (like taking the median or mean of feedback) fail. The agent falls into a "Consensus Trap"—it learns to optimize for the biased majority rather than the objective truth, a phenomenon the authors formally prove as Contextual Objective Decoupling (COD).
Methods
The authors propose CESA-LinUCB (Contextual Epistemic Source Alignment–Linear Upper Confidence Bound). Unlike traditional methods that ask "Who do I trust?", this agent asks "When do I trust this specific evaluator?"
The algorithm operates as a dual-process loop:
- Epistemic Forecasting: For every interaction, the agent uses a dedicated "Trust Model" for each evaluator to predict how likely they are to be honest in the current context.
- Trust-Weighted Action Selection: It weights the feedback from different sources based on these predictions. Sources deemed "untrustworthy" in a specific context are down-weighted.
- Sparse Axiomatic Checks: To prevent the trust models from simply agreeing with the majority, the agent occasionally (with low probability) checks the feedback against a "ground truth" axiom (an internal, verified rule). This is inspired by "Constitutional AI."
- Weighted Ridge Regression: The agent updates its main policy using only the "sanitized" data stream, effectively isolating the honest minority.
Datasets
The paper evaluates the algorithm using a newly designed "Hostile Majority" Benchmark. This is a high-dimensional contextual bandit simulation ($d=20$) where the social layer is composed of:
- Honest (20%): Provide ground truth plus noise.
- Contextual Liars (30%): Honest in one half of the context space, but invert the reward signal in the other half.
- Sycophants (50%): "Yes-men" who provide high positive feedback regardless of the action, ignoring context entirely.
This composition ensures that any method relying on simple consensus (mean, median, or majority vote) will be led astray by the 80% adversarial majority.
Key Insights and Ideas
- The Failure of Robust Statistics: One of the most surprising findings is that Median-LinUCB (a standard robust baseline) actually performed worse than a naive aggregator. Because the adversary controlled the majority, the "median" opinion was structurally captured by the liars, accelerating the agent's descent into a biased policy.
- The "Phase Transition" of Supervision: The authors found that the agent doesn't need constant supervision. There is a sharp "phase transition" where checking only 5% of interactions against a ground-truth axiom is enough to achieve near-perfect alignment. Increasing supervision beyond 5% yielded negligible marginal gains.
- Trust as a High-Dimensional Boundary: By modeling trust as a decision boundary in the context space rather than a single scalar weight, the agent can "disentangle competence from intent." It learns that an evaluator might be an expert in Context A but a "liar" in Context B.
Key Results
- Theoretical Guarantees: The authors prove that CESA-LinUCB achieves sublinear regret $\tilde{O}(\sqrt{T})$. Crucially, the "price" of dealing with adversaries scales with the complexity of the adversary's lying strategy, not the total volume of corruption.
- Empirical Recovery: In the "Hostile Majority" environment, CESA-LinUCB successfully recovered the ground truth, while all standard baselines suffered linear regret (meaning they never stopped making mistakes).
- Algorithm Agnosticism: The authors also implemented a Thompson Sampling version (CESA-TS), showing that their "Epistemic Gate" works as a universal pre-processing filter for different types of bandit algorithms.
Limitations and Future Work
- Linearity: The current work focuses on the linear bandit setting. While this provides strong theoretical foundations, real-world alignment often involves deep neural networks where the relationship between context and reward is non-linear.
- Axiom Availability: The method relies on the existence of "axiomatic checks." In very complex or subjective domains, defining a "ground truth" axiom might be as difficult as the alignment problem itself.
- Future Directions: The authors suggest extending this to Neural Social Bandits and exploring more complex social dynamics where evaluators might collude or change their strategies over time.
Summary updated