More Questions than Answers? Lessons from Integrating Explainable AI into a Cyber-AI Tool

Ashley Suh, Ashley Suh 0001, Harry Li, Caitlin Kenney, Kenneth Alperin, Steven R. GomezarXiv 2024 · DOI · arXiv

Introduction

Explainable AI (XAI) is frequently championed as the essential mechanism for establishing trust in opaque "black-box" machine learning models. In theory, frameworks such as SHAP (SHapley Additive exPlanations) and LIME (Local Interpretable Model-agnostic Explanations) provide a window into algorithmic decision-making, offering transparency that should, in principle, empower users. However, a recent study by researchers at MIT Lincoln Laboratory suggests that when these tools are deployed in the high-stakes, time-sensitive domain of cybersecurity, they often generate more confusion than clarity.

The Problem: A Mismatch in Design and Utility

Cybersecurity analysts operate in a high-pressure environment where skepticism is a fundamental requirement. For these experts to adopt AI-driven tools, they must understand the rationale behind a system's classification—for instance, why a specific block of code is flagged as malicious or why certain network traffic is deemed anomalous.

The core issue identified is a significant disconnect between the design of XAI techniques and the practical needs of domain experts. Most XAI methods are developed by AI researchers for an audience of other AI researchers. When these tools are transitioned to cybersecurity analysts—who are experts in their field but not necessarily in statistical machine learning—the resulting explanations are often "lost in translation." In a field where rapid response is critical, an explanation that requires deep statistical knowledge to interpret becomes a hindrance rather than a help.

Methodology

To investigate these challenges, the researchers conducted a case study centered on a source code classifier. The system utilized a neural network trained to analyze Python source files to determine if they implemented machine learning (ML) functionality and to identify specific sub-types, such as Decision Trees or Neural Networks.

The three-step workflow for integrating XAI into a source code classifier.

The integration followed a three-step process:

  1. Classification: The model processes Python source files to identify ML implementation.
  2. Explanation Generation: An ensemble of industry-standard local explainers, SHAP and LIME, is run on the model's output.
  3. Visualization: The interface presents "saliency maps," where specific code tokens are highlighted with colors representing their contribution to (or contradiction of) the model's prediction.

The researchers evaluated this setup through iterative demonstrations with stakeholders and representative end-users, collecting qualitative feedback on the utility and interpretability of the explanations.

Key Findings and Insights

The study revealed several critical "aha moments" that challenge current XAI philosophies:

The Problem of Contradiction

A primary source of frustration for users was the frequent disagreement between SHAP and LIME. One tool might highlight a token as strongly supporting a prediction, while the other marked the same token as contradictory. For an analyst seeking a reason to trust the AI, witnessing a conflict between the "expert" explainers significantly undermined the system's credibility.

Granularity as a Cognitive Burden

The use of token-level highlighting (e.g., coloring individual words like def, print, or import) proved to be too granular. Analysts found that performing a word-by-word audit was a "burdensome task" that disrupted their natural diagnostic workflow. They required higher-level abstractions explaining the logic of the code's behavior rather than a map of which specific tokens triggered a neural response.

Token-level saliency map showing SHAP and LIME highlights on Python source code.

The Expertise Myth

The findings challenge the assumption that SHAP and LIME are inherently accessible to "non-technical" users. In practice, interpreting these visualizations requires a high degree of "visual literacy" and an underlying grasp of feature importance concepts. Furthermore, the XAI sometimes revealed that the model relied on "silly" features (like the presence of a print statement) to make decisions. While this transparency is technically accurate, it often lowered the analysts' trust in the model's overall robustness.

Discussion and Future Directions

While users initially appreciated having some form of explanation, the specific implementation of SHAP and LIME ultimately led to distrust. To bridge this gap, the authors propose two major shifts for the future of XAI in cybersecurity:

  1. Higher-Level Abstractions: Moving beyond simple token-highlighting toward ensemble-based visualizations that can illustrate multiple factors or features simultaneously.
  2. Dialogue-Driven Systems: The researchers suggest that Large Language Models (LLMs) could serve as an interpretive bridge. Instead of static heatmaps, analysts could engage in a back-and-forth dialogue, asking questions like, "What are the shortcomings of the training data for this specific file?" This approach aligns more naturally with human workflows, though the risk of LLM "hallucinations" remains a significant hurdle.

In conclusion, while XAI holds promise, current "off-the-shelf" solutions are insufficient for the specialized needs of cybersecurity. Future development must prioritize the user's workflow and provide explanations at a level of abstraction that matches human expertise.

Summary updated

Abstract

We share observations and challenges from an ongoing effort to implement Explainable AI (XAI) in a domain-specific workflow for cybersecurity analysts. Specifically, we briefly describe a preliminary case study on the use of XAI for source code classification, where accurate assessment and timeliness are paramount. We find that the outputs of state-of-the-art saliency explanation techniques (e.g., SHAP or LIME) are lost in translation when interpreted by people with little AI expertise, despite these techniques being marketed for non-technical users. Moreover, we find that popular XAI techniques offer fewer insights for real-time human-AI workflows when they are post hoc and too localized in their explanations. Instead, we observe that cyber analysts need higher-level, easy-to-digest explanations that can offer as little disruption as possible to their workflows. We outline unaddressed gaps in practical and effective XAI, then touch on how emerging technologies like Large Language Models (LLMs) could mitigate these existing obstacles.
BibTeX
@misc{suh2024more,
  author = {Ashley Suh and Ashley Suh and Harry Li and Caitlin Kenney and Kenneth Alperin and Steven R. Gomez},
  title = {{More Questions than Answers? Lessons from Integrating Explainable AI into a Cyber-AI Tool}},
  year = {2024},
  eprint = {2408.04746},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.04746},
}
Figure