Behavioral Evaluation of Large Language Models Under Cognitive Biases in HAZOP Studies

Vinicius M. Müller, Vitor A. O. Silva, Sávio S. V. Vianna, Flávio V. SilvaarXiv 2026 · arXiv

This paper explores a critical intersection of AI and industrial safety: can we trust Large Language Models (LLMs) to help draft Hazard and Operability (HAZOP) studies without falling prey to the same cognitive biases that plague human experts?

TL;DR

While frontier LLMs are remarkably resilient—resisting nearly 97% of intentional "psychological nudges"—they remain vulnerable to specific anchoring and framing effects. These subtle prompt changes can trigger dangerous risk underestimations, proving that automated safety analysis still requires disciplined human-in-the-loop validation.

What Problem Does This Paper Address?

HAZOP analysis is the "gold standard" for identifying industrial risks, but it is notoriously exhausting and expensive. Because it relies on multidisciplinary expert workshops, it is highly susceptible to human cognitive biases like anchoring (clinging to the first number mentioned) or confirmation bias.

As researchers look to LLMs to automate or assist in drafting these safety worksheets, a major gap has emerged. Current evaluations of LLMs in this field focus on "lexical overlap"—essentially checking if the AI's text looks like a human's text. This paper argues that such metrics are useless for safety. Instead, we need to know if an LLM's behavior is robust: if you subtly change the prompt to include a biased suggestion, will the AI stay objective or follow the "anthropomorphic inferential trap"?

Methods

The researchers conducted a controlled behavioral experiment involving 420 trials across three frontier models: Google’s Gemini 3.8 Flash, Anthropic’s Claude Opus 5.5, and Meta’s Muse Spark 1.3.

Key aspects of their approach included:

  • Stateless Inference: All queries were single-pass at "temperature zero" to ensure results were as deterministic and reproducible as possible.
  • Bias Families: They tested five specific types of cognitive perturbations: Anchoring, Framing, Availability, Confirmation, and Hindsight.
  • Expert Validation: Technical outputs (causes, consequences, and risk ratings) were benchmarked through double-blind expert scoring.
  • Bayesian Modeling: They used a Bayesian generalized linear mixed model (GLMM) to separate the model's actual domain knowledge from the effects of the biased prompts.

Datasets

The study utilized a real-world reference system: Node 1 of a crude-oil production HAZOP (a Free Water Knock-Out separator). The authors built a custom battery of 47 cases based on documented engineering facts, creating a total of 420 response events to evaluate.

Key Insights and Ideas

  • Selective Resilience: Models were surprisingly good at rejecting overt social pressure (like a supervisor’s unsubstantiated "operator-blame" hypothesis) and irrelevant narratives (vignettes about incidents at other plants).
  • The "Low Anchor" Vulnerability: The models were most susceptible to "low anchors." If a prompt mentioned that a senior engineer previously estimated a risk as "Low," the models were more likely to downgrade their own assessment, even when the engineering facts suggested otherwise.
  • Framing Effects: The way a safeguard's reliability was described mattered. Highlighting a "95% success rate" (gain frame) led to more complacency and safeguard deferrals than highlighting a "5% failure rate" (loss frame), despite them being mathematically identical.
  • Quality vs. Accuracy: The study found that binary bias rates alone don't capture "engineering quality." There were wide disparities in how models reasoned through their answers, even when they arrived at the same (unbiased) conclusion.

Key Results

Across 420 trials, the models resisted 96.8% of cognitive manipulations. While this high resistance rendered the overall susceptibility statistically inconclusive (Hypothesis 1 and 2), the "failures" were safety-critical.

As shown in Figure 1, the posterior probability of a positive manipulation effect was highest for anchoring and hindsight, particularly in the Gemini model.

Posterior probability of a positive manipulation effect for each family and model.

A concrete example of this vulnerability is detailed in Table 2, where a "low anchor" caused Gemini to downgrade a risk from "Medium" to "Low" without technical justification.

Comparison of control and low-anchor judgments for a specific HAZOP case.

Limitations and Future Work

The authors acknowledge several bounds to their study:

  • Sample Size: Evaluating only a single reference node limits the statistical power for detecting modest effects.
  • Structural Zeroes: Because the control trials had zero biased events, the statistical baseline had to be "regularized," which can affect the precision of the odds ratios.
  • Future Directions: The researchers suggest testing "workflow-level debiasing" (like forcing the AI to generate a counter-argument to its own assessment) and benchmarking local, open-weight models that can handle confidential industrial data on-premise.

Ultimately, the paper concludes that while LLMs are powerful assistants, they are not "neutral safety arbiters" and must be used within a disciplined human-in-the-loop framework.

Summary updated

Abstract

Hazard and Operability (HAZOP) studies safeguard industrial processes against catastrophic accidents. Large language models (LLMs) increasingly assist HAZOP drafting, yet evaluations rely on lexical overlap. Because LLMs internalize human heuristics, prompts may trigger biased judgments. Whether frontier LLMs withstand cognitive perturbations in safety-critical workflows remains unknown. Here we show across 420 trials that three frontier LLMs resisted 96.8% of cognitive manipulations, rendering overall susceptibility statistically inconclusive. Contrary to assumed broad fragility, models rejected confirmation and availability cues. However, safety-critical risk downgrades emerged under low anchors and gain frames. Thus, automated hazard analysis requires disciplined human-in-the-loop validation.
BibTeX
@misc{muller2026behavioral,
  author = {Vinicius M. Müller and Vitor A. O. Silva and Sávio S. V. Vianna and Flávio V. Silva},
  title = {{Behavioral Evaluation of Large Language Models Under Cognitive Biases in HAZOP Studies}},
  year = {2026},
  eprint = {2611.10721},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.10721},
}
Figure