Introduction
While Large Language Models (LLMs) have demonstrated remarkable proficiency in following explicit instructions, their ability to truly comprehend the underlying intent remains a significant area of concern. In real-world scenarios, prompts often contain complex layers of text where some segments resemble instructions but are not intended for execution. This paper introduces IOINST, a novel benchmark designed to evaluate whether models can distinguish between the actual task and distracting, instruction-like content.
The IOINST Benchmark
The core of the IOINST task is a reverse-engineering challenge: given a specific response, the model must identify which instruction from a set of candidates originally produced it. This setup tests whether a model understands the intention behind an instruction rather than just reacting to its format.
The task structure involves three components:
- Context: A response generated by an LLM (e.g., "Dolphin and shark").
- Meta-Instruction: The primary task (e.g., "Identify the instruction that generated the response above").
- Candidate Instructions: A set of four options, including the correct "Label Instruction" and three distractors.
Methodology and Error Analysis
To rigorously evaluate model performance, the researchers categorized failures into two distinct error types:
- Case 1: The model attempts the meta-instruction but selects the incorrect candidate.
- Case 2: The model fails to follow the meta-instruction entirely, instead executing one of the candidate instructions (e.g., naming animals instead of selecting the text of the instruction).
Distractors were crafted with varying levels of difficulty using a pipeline involving semantic search and human verification to ensure the "Context" response maps uniquely to the label instruction.
Distractor types include:
- Random: Unrelated instructions.
- Semantic: Instructions semantically similar to the label but leading to different outputs.
- Anti-Attribute: Instructions sharing the same task but differing in fine-grained constraints (e.g., word count).
Key Insights
The Instruction Distraction Effect
LLMs exhibit a "reflexive" tendency to execute any command-like text they encounter. This is evidenced by the high frequency of Case 2 errors, where models ignore the meta-instruction to engage with distractors. This suggests that instruction-tuning may have over-sensitized models to specific formats at the expense of deep comprehension.
The Few-Shot Paradox
Counter-intuitively, providing few-shot examples often degraded performance on the IOINST benchmark. The authors suggest that these examples introduce additional instruction-formatted noise, further confusing the model's perception of the primary intent.
Structural Sensitivity
Model performance improved when meta-instructions were detailed and when the "Context" (the response) was presented before the candidate instructions, indicating that prompt engineering remains a critical factor in intent recognition.
Results
Experimental results highlight a stark contrast between closed-source and open-source models. While GPT-4o achieved approximately 95% accuracy in simpler settings, open-source models like Llama-3-8B struggled significantly, particularly in the Anti-Attribute category where accuracy plummeted to 23.58%. Furthermore, increasing the model's "temperature" (randomness) led to a sharp decline in the ability to follow meta-instructions, making models more susceptible to distraction.
Conclusion
The IOINST benchmark reveals that current LLMs, while capable of following isolated commands, lack robust instruction comprehension in complex contexts. Future work should focus on data-centric training strategies and architectural improvements to help models prioritize user intent over superficial formatting.
Summary updated