HIPO: Instruction Hierarchy via Constrained Reinforcement Learning

Keru Chen, Jun Luo, Sen Lin, Yingbin Liang, Alvaro Velasquez, Nathaniel D. Bastian, Shaofeng Zou

Abstract

Hierarchical Instruction Following (HIF) refers to the problem of prompting large language models with a priority-ordered stack of instructions. Standard methods like RLHF and DPO typically fail in this problem since they mainly optimize for a single objective, failing to explicitly enforce system prompt compliance. Meanwhile, supervised fine-tuning relies on mimicking filtered, compliant data, which fails to establish the priority asymmetry at the algorithmic level. In this paper, we introduce HIPO, a novel alignment framework that formulates HIF as a Constrained Markov Decision Process. HIPO elevates system prompts from mere input context to strict algorithmic boundaries. Using a primal-dual safe reinforcement learning approach, the algorithm dynamically enforces system prompt compliance as an explicit constraint, maximizing user utility strictly within this feasible region. Extensive evaluations across diverse model architectures (e.g., Qwen, Phi, Llama) demonstrate that HIPO significantly improves both system compliance and user utility. Furthermore, mechanistic analysis reveals that this constrained optimization autonomously drives the model to shift its attention toward long-range system tokens, providing a principled foundation for reliable LLM deployment in complex workflows.

Introduction

Large Language Models (LLMs) often face a "tug-of-war" between two masters: the system prompt (which defines global rules, safety boundaries, and personas) and the user prompt (the immediate task). In complex workflows, these instructions often conflict. For example, a system prompt might forbid direct answers to encourage pedagogical inquiry, but a user might demand a direct list of facts.

Current alignment methods like RLHF (Reinforcement Learning from Human Feedback) or DPO (Direct Preference Optimization) typically collapse these instructions into a single objective. This "flat" approach fails to establish a clear hierarchy; the model often prioritizes the user's immediate request at the expense of the system's constraints, leading to "jailbreaks" or rule-breaking. Supervised Fine-Tuning (SFT) also struggles because it simply mimics data rather than learning the underlying algorithmic priority of the system prompt.

Methodology: The HIPO Framework

The authors introduce HIPO (Hierarchical Instruction Policy Optimization), which reframes Hierarchical Instruction Following (HIF) as a Constrained Markov Decision Process (CMDP). Instead of a single reward, HIPO uses two distinct signals: System Compliance ($r_{sys}$) and User Utility ($r_{user}$).

The HIPO framework: (A) Problem setting for hierarchical instruction following, (B) Pareto improvement over standard methods, and (C) The training pipeline using primal-dual optimization.

Primal-Dual Optimization

HIPO treats system compliance as a hard constraint (e.g., "Compliance must be at least $\tau$"). It uses a primal-dual safe RL approach:

  • The Primal Step: Updates the model policy $\pi$ to maximize user utility.
  • The Dual Step: Dynamically updates a Lagrange multiplier ($\lambda$). If the model starts violating system constraints, $\lambda$ increases, heavily penalizing the model until it returns to the "feasible region" of compliance.

To keep training efficient, HIPO builds on GRPO (Group Relative Policy Optimization), which uses group-based sampling to estimate advantages without needing a separate, memory-heavy "critic" model.

Experimental Setup

The researchers evaluated HIPO using the SystemCheck dataset, sampling 2,000 hierarchical instruction pairs with a strict 1:1 ratio of Aligned instances (where user requests naturally fit system rules) and Conflicting instances (where the user request explicitly tries to bait the model into breaking a system rule).

Evaluation was performed using a decoupled LLM-as-a-Judge protocol, where the judge evaluates system compliance and user utility in two separate passes to avoid "aspect interference."

Results and Analysis

The authors tested HIPO across several model families (Qwen, Phi, Llama) and sizes (1.7B to 8B parameters).

Performance comparison across various LLMs on the SystemCheck dataset, showing system compliance and user utility scores (Compliance / Utility).

Key Findings

  • Superior Compliance: On the Qwen-1.7B model, HIPO reached the target system compliance threshold of 0.7, while SFT and DPO hovered around 0.6 in conflict scenarios.
  • Higher Utility: While "Sys-only" models (optimized only for rules) became overly conservative and useless, HIPO maintained significantly higher user utility scores (e.g., 0.47 vs 0.22 for Sys-only on Qwen-1.7B).
  • Breaking the Pareto Trade-off: HIPO pushes the Pareto frontier, achieving higher user utility at the same level of system compliance compared to SFT or DPO. Mechanistic analysis reveals this is driven by the model autonomously learning to shift attention toward long-range system tokens at the start of a response.

Conclusion

HIPO provides a principled foundation for reliable LLM deployment by enforcing instruction hierarchy at the algorithmic level. By treating system prompts as strict boundaries rather than mere context, it effectively mitigates the "identity crisis" often seen in complex instruction-following tasks. Future work will focus on reducing judge dependency and improving robustness in adversarial out-of-distribution scenarios.

Summary updated

Abstract

Hierarchical Instruction Following (HIF) refers to the problem of prompting large language models with a priority-ordered stack of instructions. Standard methods like RLHF and DPO typically fail in this problem since they mainly optimize for a single objective, failing to explicitly enforce system prompt compliance. Meanwhile, supervised fine-tuning relies on mimicking filtered, compliant data, which fails to establish the priority asymmetry at the algorithmic level. In this paper, we introduce \textsc{HIPO}, a novel alignment framework that formulates HIF as a Constrained Markov Decision Process. \textsc{HIPO} elevates system prompts from mere input context to strict algorithmic boundaries. Using a primal-dual safe reinforcement learning approach, the algorithm dynamically enforces system prompt compliance as an explicit constraint, maximizing user utility strictly within this feasible region. Extensive evaluations across diverse model architectures (e.g., Qwen, Phi, Llama) demonstrate that \textsc{HIPO} significantly improves both system compliance and user utility. Furthermore, mechanistic analysis reveals that this constrained optimization autonomously drives the model to shift its attention toward long-range system tokens, providing a principled foundation for reliable LLM deployment in complex workflows.
BibTeX
@misc{chen2026hipo,
  author = {Keru Chen and Jun Luo and Sen Lin and Yingbin Liang and Alvaro Velasquez and Nathaniel D. Bastian and Shaofeng Zou},
  title = {{HIPO: Instruction Hierarchy via Constrained Reinforcement Learning}},
  year = {2026},
  eprint = {2603.16152},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.16152},
}
Figure