TL;DR
Large Language Models (LLMs) often struggle to prioritize system instructions over social cues, leading to unpredictable behavior. This paper investigates how LLMs internally represent and resolve these conflicts, finding that system-user conflicts are detected but not consistently resolved, while social cues lead to more reliable obedience. The work highlights the fragility of system obedience and suggests a need for more robust alignment methods.
What Problem Does This Paper Address?
LLMs are designed with a hierarchical instruction structure where system prompts (e.g., "You are a helpful assistant") are meant to take precedence over user prompts. However, empirical studies have shown that LLMs frequently disregard this hierarchy, often prioritizing social cues like authority or consensus embedded in natural language over explicit system instructions. This "fragile system obedience" poses a significant problem for safety-critical applications and makes LLMs susceptible to manipulation. This paper aims to understand the underlying mechanisms within LLMs that lead to this behavior, specifically by examining how conflicts between system and user instructions, and between different social cues, are represented and resolved in the model's internal states.
Methods
The researchers employed a multi-pronged approach to dissect LLM behavior in conflict scenarios:
- Linear Probing: This technique was used to identify where in the model's internal activations (hidden states) the decision about which instruction to follow is encoded. By training simple linear classifiers on these activations, they could determine if conflict-decision signals are localized and how they differ across various conflict types (system-user vs. social cues).
- Logit Attribution (LA): To understand how conflicting instructions compete, LA was used to attribute the model's next-token prediction (logit) to specific parts of the input, namely the tokens corresponding to the conflicting instructions. This allowed them to quantify the influence of each instruction on the final decision.
- Steering Vectors: This method involves intervening directly in the model's internal representations. By identifying the difference between representations associated with social cues and system instructions, they created a "steering vector." Injecting this vector into the model's activations during inference aims to shift its behavior towards prioritizing system instructions.
Datasets
The study utilized an augmented version of a dataset from a previous work ([6]) that contained explicitly conflicting instructions. This dataset was expanded to include five constraint types: Language, Word Count, Sentence Count, Keyword Usage (include/exclude), and Keyword Frequency. For each category, 30 systematic variations were created and combined with four role-conflict framings (system-user and three social hierarchy types: Organizational Authority, Expertise Credibility, and Social Consensus), resulting in a total of 120,000 prompts.
Key Insights and Ideas
- Distinct Representation Subspaces: Conflict-decision signals for system-user conflicts and social-role conflicts are encoded in largely distinct, orthogonal subspaces within the LLM's hidden states. This suggests that the model "sees" these types of conflicts differently.
- Early Conflict Detection, Uneven Resolution: LLMs detect system-user conflicts early in their processing layers, but they struggle to consistently resolve them in favor of the system prompt. In contrast, social cues lead to more robust and consistent obedience.
- Social Cues as "Super-Bias" Signals: Social cues act as powerful "super-bias" signals that can suppress the resolution of conflicts, potentially leading to fairness concerns and making models more susceptible to manipulation.
- Role-Agnostic Steering: While steering vectors derived from social cues can amplify instruction following, this amplification is "role-agnostic." It pushes the model towards a target behavior (e.g., using a keyword more often) regardless of whether the system or user issued that instruction. This means it doesn't necessarily reinforce the intended hierarchy.
- Random Steering Can Also Influence Behavior: Surprisingly, even random steering vectors, when scaled sufficiently, can increase instruction-following behavior. This suggests that simply amplifying certain internal features can bias the model towards stronger compliance, highlighting the sensitivity of LLMs to internal state manipulations.
Key Results
- Linear Probing: Probe performance for conflict decisions rises sharply in early layers, peaking around layer 10, indicating that the model quickly forms an internal representation of its obedience decision. All tested positions (attention output, MLP output, post-MLP residual stream) showed high AUC (> 0.89).
- Logit Attribution: System-user conflicts showed a higher rate of "opposing signed contributions" (25.61%) compared to social consensus conflicts (16.42%), indicating more internal competition. However, for social consensus conflicts, the primary role reliably won when conflict was detected, whereas system-user conflicts were less consistently resolved.
- Steering Experiments: Steering vectors, when applied, reliably amplified instruction following. For example, in word count conflicts, increasing steering strength could shift the output towards the system's constraint (e.g., shorter responses), but at the cost of fluency and stability. In keyword count conflicts, steering could enforce a high keyword count, but this was role-agnostic and could lead to repetitive and nonsensical outputs.
Limitations and Future Work
- Fragile System Obedience: The paper confirms that system instructions remain fragile, especially in open-weight models without additional training. This makes LLMs vulnerable to prompt-injection attacks framed with authoritative language.
- Role-Agnostic Steering: The current steering method, while amplifying instruction following, does not differentiate between system and user roles. This means it doesn't effectively reinforce the intended hierarchy.
- Future Directions:
- Developing role-agnostic steering methods that can selectively boost attention on system instructions while suppressing user instructions.
- Exploring training-based methods that create cleaner, hierarchy-sensitive vectors, potentially by using specialized datasets.
- Investigating lightweight alternatives that learn mappings between contrastive subspaces (e.g., system-user vs. social roles) while preserving conflict detection. The authors note that system-user conflicts show strong internal opposition, which might be lost if alignment simply overwrites signals.
Summary updated