Order Matters: Investigate the Position Bias in Multi-constraint Instruction Following

Jie Zeng, Qianyu He, Qingyu Ren, Jiaqing Liang, Yanghua Xiao, Weikang Zhou, Zeye Sun, Fei Yu

Introduction

Large Language Models (LLMs) have demonstrated remarkable capabilities in following complex instructions. However, their performance remains notoriously sensitive to prompt phrasing. While previous research has highlighted "lost-in-the-middle" phenomena regarding long-context retrieval, this study investigates a more granular "position bias": how the specific order of multiple constraints within a single prompt affects a model's ability to satisfy them all. By introducing the Constraint Difficulty Distribution Index (CDDI), this research demonstrates that the sequence of instructions—specifically moving from "hard" to "easy" constraints—is a critical factor in model reliability.

The Problem: Internal Sequence Sensitivity

As AI agents evolve to handle more sophisticated tasks, users frequently provide "multi-constraint" instructions that require the model to simultaneously manage formatting, stylistic, length, and content requirements (e.g., "Write a summary in JSON format, using no commas, and keeping it under 50 words"). Currently, when an LLM fails such a prompt, users often resort to trial-and-error reordering. This paper moves beyond anecdotal prompt engineering to systematically explore whether a universal optimal order exists across different model architectures.

Methodology

To quantify position bias, the researchers developed a systematic probing framework consisting of three main components:

1. Difficulty Measurement

Rather than assigning difficulty subjectively, the authors measured the average accuracy of various LLMs (including LLaMA3, Mistral, and GPT-4o-mini) across specific constraint types. Constraints that models frequently failed (like "No commas") were labeled as "hard," while those with high success rates (like "JSON format") were labeled as "easy."

2. The CDDI Metric

The Constraint Difficulty Distribution Index (CDDI) was proposed to quantify the arrangement of a prompt. Using the Kendall tau distance, the metric scores a sequence from -1 (Easy-to-Hard) to 1 (Hard-to-Easy). This allows for a mathematical correlation between the "slope" of difficulty in a prompt and the resulting model performance.

3. Feature Attribution

To understand the underlying mechanics, the authors utilized gradient-based methods to visualize the "importance weights" assigned to different parts of the prompt during the generation process. This helped determine if the model's attention was being distributed differently based on the instruction order.

The PBIF Dataset

The study introduced the PBIF dataset, a comprehensive collection of 24,000 samples synthesized from Natural Instructions V2, Self-Instruct, and Open Assistant. Constraints were categorized into eight distinct groups:

  • Keyword: Specific words to include/exclude.
  • Language: Target output language.
  • Length: Word or sentence counts.
  • Content: Specific themes or topics.
  • Format: JSON, Markdown, etc.
  • ChangeCase: Uppercase/lowercase requirements.
  • StartEnd: Specific starting or ending phrases.
  • Punctuation: Restrictions on specific symbols.

Key Insights

The "Hard-to-Easy" Advantage

Across almost all tested models, the Hard-to-Easy (CDDI=1) order yielded the highest accuracy. Front-loading the most difficult constraints appears to prime the model to prioritize the requirements it is most likely to forget or fail.

The Attention Ladder

Gradient analysis revealed that a Hard-to-Easy distribution creates an "iterative, laddered" attention pattern. Instead of the model's focus fading or staying static, this ordering helps the model maintain a high level of attention on the constraint requirements throughout the entire generation process.

Multi-round Fragility

Position bias is significantly more pronounced in multi-round conversations. If a difficult constraint is introduced late in a dialogue, the model is much more likely to ignore it. In LLaMA3-8B, the performance gap between the best and worst orders grew from 7% in single-round tasks to a staggering 25% in multi-round settings.

Results and Performance Impact

  • LLaMA3-8B: Showed significant sensitivity, with a 25% improvement in multi-round settings simply by reordering constraints from hard-to-easy.
  • GPT-4o-mini: While generally more robust and stable, it still exhibited a clear preference for the hard-to-easy distribution, suggesting that position bias is likely inherent to the Transformer architecture or common pre-training patterns.
  • Constraint Sensitivity: "Length" constraints were found to be particularly sensitive; if placed late in a prompt, models struggled to adjust the output length effectively.

Limitations and Future Work

The study focused on "parallel" constraints—those that are logically independent. It did not explore "sequential" constraints where one step must follow another (e.g., "Step 1: Summarize, Step 2: Translate"). Additionally, while the paper identifies a successful prompting strategy (putting the hard stuff first), it does not yet provide a training-based mitigation to make models truly order-agnostic.

Future research may focus on developing "order-robust" training techniques to ensure that LLMs remain performant regardless of how a user sequences their requirements, reducing the burden of prompt engineering on the end-user.

Summary updated

Abstract

Real-world instructions with multiple constraints pose a significant challenge to existing large language models (LLMs). An observation is that the LLMs exhibit dramatic performance fluctuation when disturbing the order of the incorporated constraints. Yet, none of the existing works has systematically investigated this position bias problem in the field of multi-constraint instruction following. To bridge this gap, we design a probing task where we quantitatively measure the difficulty distribution of the constraints by a novel Difficulty Distribution Index (CDDI). Through the experimental results, we find that LLMs are more performant when presented with the constraints in a ``hard-to-easy'' order. This preference can be generalized to LLMs with different architecture or different sizes of parameters. Additionally, we conduct an explanation study, providing an intuitive insight into the correlation between the LLM's attention and constraint orders. Our code and dataset are publicly available at https://github.com/meowpass/PBIF.
BibTeX
@inproceedings{zeng2025order,
  author = {Jie Zeng and Qianyu He and Qingyu Ren and Jiaqing Liang and Yanghua Xiao and Weikang Zhou and Zeye Sun and Fei Yu},
  title = {{Order Matters: Investigate the Position Bias in Multi-constraint Instruction Following}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure