Are Dilemmas and Conflicts in LLM Alignment Solvable? A View from Priority Graph

Zhenheng Tang, Xiang Liu, Qian Wang, Eunsol Choi, Bo Li, Xiaowen ChuarXiv 2026 · arXiv

TL;DR

This paper introduces the Priority Graph, a formal framework to model how Large Language Models (LLMs) navigate conflicting instructions and values across different contexts. The authors expose a critical vulnerability called "priority hacking"—where safety filters are bypassed by appealing to higher-order values like justice—and propose a runtime verification mechanism to ground LLM contexts in real-world data.

What Problem Does This Paper Address?

As LLMs transition from simple text generators to autonomous agents, they increasingly face situations where instructions, internal knowledge, and ethical values clash. Traditional alignment methods often treat safety and helpfulness as static, linear hierarchies—reminiscent of Isaac Asimov’s "Three Laws of Robotics."

However, human values are context-dependent and pluralistic. Current models struggle when a system prompt contradicts a user prompt, or when internal training data clashes with real-time retrieved information. This paper addresses the lack of a unified framework to taxonomize, formalize, and mitigate these complex, dynamic conflicts in LLM alignment.

Figure 1: Five different types of conflicts of current LLM applications and usages.

Methods

The authors tackle this problem through a mix of conceptual taxonomy and graph theory:

  • Taxonomy of Conflicts: They categorize LLM alignment conflicts into five distinct types: Instruction Conflicts, Information Conflicts (e.g., RAG trust dilemmas), Ethics Dilemmas (e.g., the Trolley Problem), Value Dilemmas (e.g., Truthfulness vs. Protection), and Preference Dilemmas (e.g., subjective aesthetic judgments).
  • Priority Graph ($G_C$): They formalize these relationships using a context-dependent directed graph. Nodes represent instructions or values, while directed edges represent priority relationships determined by the model's conditional output probability distribution ($p_\theta(D|A_1, A_2, C)$). If a model chooses action $A_1$ over $A_2$ in context $C$, a directed edge points from $A_1$ to $A_2$.
  • Runtime Verification Mechanism: To defend against context manipulation, the authors propose a defense layer where the LLM actively queries external, trustworthy information sources to validate the premises of a user's prompt before executing potentially risky instructions.

Datasets

This is a theoretical framework and position paper; it does not introduce a new empirical dataset or conduct quantitative benchmark evaluations. It references existing datasets, such as the DailyDilemmas dataset, to illustrate specific value conflicts, but no new training or evaluation datasets were generated for this work.

Key Insights and Ideas

  • The Graph is Dynamic and Cyclical: Unlike rigid rule systems, an LLM's priority graph dynamically reconfigures based on conversational history, user preferences, and external environments. Crucially, these graphs can contain directed cycles (e.g., $A_1 \succ A_2 \succ A_3 \succ A_1$), representing irreconcilable logical paradoxes.
  • Priority Hacking: The authors reveal that safety alignment can be undermined by weaponizing virtue. An adversary can craft a deceptive context ($C_{adv}$) that frames a malicious request (like writing a phishing email) as an essential act in service of a higher-priority benign value like "Justice" (e.g., pretending to be an investigative journalist exposing corporate crime).
The priority graph framework, priority hacking, and the proposed runtime verification defense.
  • Philosophical Irreducibility: While technical solutions like runtime verification can resolve factual deceptions, deep ethical dilemmas (e.g., Utilitarianism vs. Deontology) are fundamentally unresolvable by code or data alignment alone.

Key Results

Because this is a conceptual framework and position paper, quantitative experimental results (such as accuracy or speedup metrics) are not discussed in the paper.

Limitations and Future Work

The primary limitation acknowledged by the authors is that technical alignment cannot "solve" philosophically irreducible value conflicts where no universal human consensus exists.

They highlight several open questions for future research:

  1. Model Behavior: How should an LLM behave when faced with an unresolvable ethical conflict? Should it refuse to answer, present multiple philosophical perspectives, or allow the end-user to explicitly steer its core value priorities?
  2. Implementation Challenges: The practical implementation, latency overhead, and security of the proposed runtime verification mechanism remain to be fully developed and tested in real-world deployment scenarios.

Summary updated

Abstract

As Large Language Models (LLMs) become more powerful and autonomous, they increasingly face conflicts and dilemmas in many scenarios. We first summarize and taxonomize these diverse conflicts. Then, we model the LLM's preferences to make different choices as a priority graph, where instructions and values are nodes, and the edges represent context-specific priorities determined by the model's output distribution. This graph reveals that a unified stable LLM alignment is very challenging, because the graph is neither static nor necessarily consistent in different contexts. Besides, it also reveals a potential vulnerability: priority hacking, where adversaries can craft deceptive contexts to manipulate the graph and bypass safety alignments. To counter this, we propose a runtime verification mechanism, enabling LLMs to query external sources to ground their context and resist manipulation. While this approach enhances robustness, we also acknowledge that many ethical and value dilemmas are philosophically irreducible, posing a long-term, open challenge for the future of AI alignment.
BibTeX
@misc{tang2026are,
  author = {Zhenheng Tang and Xiang Liu and Qian Wang and Eunsol Choi and Bo Li and Xiaowen Chu},
  title = {{Are Dilemmas and Conflicts in LLM Alignment Solvable? A View from Priority Graph}},
  year = {2026},
  eprint = {2603.15527},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.15527},
}
Figure