This paper explores a critical intersection of AI and industrial safety: can we trust Large Language Models (LLMs) to help draft Hazard and Operability (HAZOP) studies without falling prey to the same cognitive biases that plague human experts?
TL;DR
While frontier LLMs are remarkably resilient—resisting nearly 97% of intentional "psychological nudges"—they remain vulnerable to specific anchoring and framing effects. These subtle prompt changes can trigger dangerous risk underestimations, proving that automated safety analysis still requires disciplined human-in-the-loop validation.
What Problem Does This Paper Address?
HAZOP analysis is the "gold standard" for identifying industrial risks, but it is notoriously exhausting and expensive. Because it relies on multidisciplinary expert workshops, it is highly susceptible to human cognitive biases like anchoring (clinging to the first number mentioned) or confirmation bias.
As researchers look to LLMs to automate or assist in drafting these safety worksheets, a major gap has emerged. Current evaluations of LLMs in this field focus on "lexical overlap"—essentially checking if the AI's text looks like a human's text. This paper argues that such metrics are useless for safety. Instead, we need to know if an LLM's behavior is robust: if you subtly change the prompt to include a biased suggestion, will the AI stay objective or follow the "anthropomorphic inferential trap"?
Methods
The researchers conducted a controlled behavioral experiment involving 420 trials across three frontier models: Google’s Gemini 3.8 Flash, Anthropic’s Claude Opus 5.5, and Meta’s Muse Spark 1.3.
Key aspects of their approach included:
- Stateless Inference: All queries were single-pass at "temperature zero" to ensure results were as deterministic and reproducible as possible.
- Bias Families: They tested five specific types of cognitive perturbations: Anchoring, Framing, Availability, Confirmation, and Hindsight.
- Expert Validation: Technical outputs (causes, consequences, and risk ratings) were benchmarked through double-blind expert scoring.
- Bayesian Modeling: They used a Bayesian generalized linear mixed model (GLMM) to separate the model's actual domain knowledge from the effects of the biased prompts.
Datasets
The study utilized a real-world reference system: Node 1 of a crude-oil production HAZOP (a Free Water Knock-Out separator). The authors built a custom battery of 47 cases based on documented engineering facts, creating a total of 420 response events to evaluate.
Key Insights and Ideas
- Selective Resilience: Models were surprisingly good at rejecting overt social pressure (like a supervisor’s unsubstantiated "operator-blame" hypothesis) and irrelevant narratives (vignettes about incidents at other plants).
- The "Low Anchor" Vulnerability: The models were most susceptible to "low anchors." If a prompt mentioned that a senior engineer previously estimated a risk as "Low," the models were more likely to downgrade their own assessment, even when the engineering facts suggested otherwise.
- Framing Effects: The way a safeguard's reliability was described mattered. Highlighting a "95% success rate" (gain frame) led to more complacency and safeguard deferrals than highlighting a "5% failure rate" (loss frame), despite them being mathematically identical.
- Quality vs. Accuracy: The study found that binary bias rates alone don't capture "engineering quality." There were wide disparities in how models reasoned through their answers, even when they arrived at the same (unbiased) conclusion.
Key Results
Across 420 trials, the models resisted 96.8% of cognitive manipulations. While this high resistance rendered the overall susceptibility statistically inconclusive (Hypothesis 1 and 2), the "failures" were safety-critical.
As shown in Figure 1, the posterior probability of a positive manipulation effect was highest for anchoring and hindsight, particularly in the Gemini model.
A concrete example of this vulnerability is detailed in Table 2, where a "low anchor" caused Gemini to downgrade a risk from "Medium" to "Low" without technical justification.
Limitations and Future Work
The authors acknowledge several bounds to their study:
- Sample Size: Evaluating only a single reference node limits the statistical power for detecting modest effects.
- Structural Zeroes: Because the control trials had zero biased events, the statistical baseline had to be "regularized," which can affect the precision of the odds ratios.
- Future Directions: The researchers suggest testing "workflow-level debiasing" (like forcing the AI to generate a counter-argument to its own assessment) and benchmarking local, open-weight models that can handle confidential industrial data on-premise.
Ultimately, the paper concludes that while LLMs are powerful assistants, they are not "neutral safety arbiters" and must be used within a disciplined human-in-the-loop framework.
Summary updated