Too Much to Trust? Measuring the Security and Cognitive Impacts of Explainability in AI-Driven SOCs

Nidhi Rastogi, Shirid Pant, Devang Dhanuka, Abhinav Saxena, Pranjal Mairal

TL;DR

This paper identifies a major "usability gap" in how Explainable AI (XAI) is used in Security Operations Centers (SOCs), finding that generic model-internal explanations often overwhelm analysts. To fix this, the authors introduce CORTEX, a framework that dynamically tailors AI explanations based on an analyst’s specific role, the severity of the threat, and the need for "attack narratives" rather than isolated data points.

What Problem Does This Paper Address?

Modern Security Operations Centers (SOCs) are drowning in data. Analysts face thousands of alerts daily, leading to "alert fatigue"—a state of cognitive overload where real threats are easily missed. While AI-driven detection helps flag these threats, the models are often "black boxes."

Existing Explainable AI (XAI) techniques (like SHAP or LIME) were largely designed by data scientists for debugging models. However, a Tier-1 security analyst under high pressure doesn't need a list of feature weights; they need to know if an alert is a false positive and what to do next. The authors address the lack of human-centered XAI that accounts for the high-stakes, time-sensitive, and role-stratified nature of cybersecurity work.

Methods

The researchers employed a three-month, mixed-methods approach to bridge the gap between AI theory and SOC reality:

  1. Online Survey (N=248): Captured broad quantitative data on how SOC professionals across four continents interact with current tools and their attitudes toward XAI.
  2. In-depth Interviews (N=24): Conducted 810 minutes of interviews to explore the "why" behind analyst preferences, using synthetic LLM-generated explanations to test different formats.
  3. CORTEX Framework: Based on these insights, they developed the COntextual Role and Trust aware EXplanation framework. CORTEX uses four modules (User Profiler, Context Engine, Explanation Generator, and Feedback Module) to provide "progressive disclosure"—starting with a high-level summary and allowing analysts to drill down into technical details only when necessary.

Datasets

As a human-centric study, the primary "datasets" were the qualitative and quantitative responses from the 248 survey participants and 24 interviewees. To evaluate the CORTEX framework, the authors used:

  • Synthetic Alert Logs: Created using OpenAI’s GPT-4 to simulate realistic security scenarios (e.g., phishing, brute-force attacks).
  • Simulated SOC Environment: A controlled setup where analysts triaged 10 alerts to compare a baseline interface against the CORTEX-enhanced interface.

Key Insights and Ideas

  • The "Trust-Explainability Curve": The authors found that trust does not increase linearly with more information. Instead, there is a "sweet spot." Too little information breeds skepticism, but too much detail leads to the "Overload Zone," where "explanation friction" actually slows down the investigation.
  • Action over Internals: Analysts consistently prioritized actionable insights (e.g., "this IP is on a known blacklist") over model internals (e.g., "feature X contributed 0.2 to the score").
  • Role-Based Needs: A Tier-1 analyst (triage) needs brevity and "next-step" recommendations. A Tier-3 analyst (threat hunter) needs the raw logs and model provenance to conduct a deep-dive forensic analysis.
  • Narrative vs. Snapshot: AI usually explains a single event. Analysts, however, think in "attack narratives." They want to see how a failed login at 10:00 AM connects to a data transfer at 10:05 AM.

Key Results

The study’s evaluation of the CORTEX framework yielded impressive improvements in a simulated SOC environment:

  • Efficiency: Average triage time per alert dropped from 4.1 minutes to 3.3 minutes (a ~20% speedup).
  • Cognitive Load: Using the NASA-TLX scale, the researchers measured an 18% decrease in perceived cognitive load for analysts using CORTEX.
  • Trust Calibration: Analysts using CORTEX aligned their trust with the model’s actual confidence 82% of the time, compared to only 64% in the baseline group. This suggests that better explanations help analysts know exactly when to trust the AI and when to be skeptical.

Limitations and Future Work

The authors acknowledge that their evaluation was conducted in a simulated environment with synthetic logs, which may not fully capture the chaos of a real-world SOC. Additionally, the study relied on a relatively small interview sample (N=24), though it was geographically diverse.

Future Work includes:

  • Integrating CORTEX into live, production-grade SIEM/XDR platforms.
  • Automating the generation of "Multi-Stage Attack Narratives" using graph-based clustering.
  • Studying the long-term effects of XAI on "automation bias"—the tendency for humans to over-rely on automated systems even when they are wrong.

Summary updated

Abstract

Explainable AI (XAI) holds significant promise for enhancing the transparency and trustworthiness of AI-driven threat detection in Security Operations Centers (SOCs). However, identifying the appropriate level and format of explanation, particularly in environments that demand rapid decision-making under high-stakes conditions, remains a complex and underexplored challenge. To address this gap, we conducted a three-month mixed-methods study combining an online survey (N1=248) with in-depth interviews (N2=24) to examine (1) how SOC analysts conceptualize AI-generated explanations and (2) which types of explanations are perceived as actionable and trustworthy across different analyst roles. Our findings reveal that participants were consistently willing to accept XAI outputs, even in cases of lower predictive accuracy, when explanations were perceived as relevant and evidence-backed. Analysts repeatedly emphasized the importance of understanding the rationale behind AI decisions, expressing a strong preference for contextual depth over a mere presentation of outcomes on dashboards. Building on these insights, this study re-evaluates current explanation methods within security contexts and demonstrates that role-aware, context-rich XAI designs aligned with SOC workflows can substantially improve practical utility. Such tailored explainability enhances analyst comprehension, increases triage efficiency, and supports more confident responses to evolving threats.
BibTeX
@misc{rastogi2025too,
  author = {Nidhi Rastogi and Shirid Pant and Devang Dhanuka and Abhinav Saxena and Pranjal Mairal},
  title = {{Too Much to Trust? Measuring the Security and Cognitive Impacts of Explainability in AI-Driven SOCs}},
  year = {2025},
  eprint = {2503.02065},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.02065},
}
Figure