TL;DR
This paper introduces a novel computer-aided method for capturing the intricate cognitive processes cyber-attack analysts use. By integrating automated data logging with situated self-reports, it aims to provide a more unobtrusive and comprehensive understanding of how analysts reason through complex threats, paving the way for better training and tool development.
What Problem Does This Paper Address?
Cyber-attacks are becoming increasingly sophisticated, demanding that analysts process vast amounts of data and reason under uncertainty to detect and understand threats. While understanding the cognitive processes of these analysts is crucial for improving their performance, existing methods for Cognitive Task Analysis (CTA) are often intrusive, time-consuming, or rely heavily on self-reporting, which can be unreliable. Specifically, cyber-attack analysis is a "human-in-the-loop" process that requires analysts to sift through enormous datasets, identify subtle patterns, and connect disparate pieces of evidence to form hypotheses. This high cognitive load and the sensitive nature of real-world network data (confidentiality concerns) make traditional CTA methods challenging to apply effectively.
Methods
The authors propose an "integrated computer-aided data collection method" for CTA, built on three core components:
- Trace Representation: A formal definition of a "cognitive trace" that captures the analytical reasoning process as a sequence of operations (actions, observations, hypotheses) over time, along with their context and connections.
- ARSCA (Analytical Reasoning Support system for Cyber Analysis) Tool: A custom Windows application designed to support this trace representation. ARSCA integrates:
- Automated Capture: It unobtrusively records analyst actions like searching, filtering, and data selection.
-
Situated Self-Reports: It allows analysts to input their hypotheses and observations directly within the tool, linking them to the captured data. This integration aims to minimize distraction and workload by having the tool capture actions automatically while providing a structured way for analysts to articulate their thoughts.
-
Laboratory Experiment: A controlled experiment where participants use ARSCA to perform a simulated cyber-attack analysis task. This approach isolates the experiment from real organizational networks, addressing confidentiality concerns.
Datasets
The study utilized a portion of the network dataset from the VAST Challenge 2012 Mini Challenge 2. To make the dataset manageable for the experiment, a critical 10-minute period containing three attack events was extracted. This subset included 239 IDS alerts and 115,524 firewall logs. The ground truth of the original attack events was used to evaluate participant performance.
Key Insights and Ideas
- The Power of Integration: Combining automated capture with situated self-reports in ARSCA significantly enhances the richness and reliability of CTA data. Automated capture provides objective actions, while self-reports offer the crucial "why" behind those actions.
- Capturing Implicit Reasoning: While traces may not capture every fleeting thought, the sequence of operations and the explicit hypotheses recorded can provide strong clues and enable researchers to infer implicit reasoning processes and domain knowledge.
- Complementary Data: Automatically captured actions provide context for observations, and self-reported hypotheses explain the motivation for subsequent actions, demonstrating how different data types within the trace mutually reinforce understanding.
- Anchoring Bias in Action: The motivating example highlights how analysts can fall prey to cognitive biases like anchoring, underscoring the need for methods that can help analysts reflect on and potentially overcome such pitfalls.
- Experience Matters: The case studies demonstrate how experienced analysts leverage domain knowledge and pattern recognition to identify subtle attack patterns and infer associated events, a process that is difficult to capture without detailed trace analysis.
Key Results
- The preliminary trace analysis showed highly promising results, indicating the feasibility of the proposed method.
- The ARSCA tool successfully captured a balanced number of automated items (151) and self-reports (180) across 30 traces.
- A Wilcoxon signed-rank test confirmed the accuracy of ARSCA's timestamps compared to a standard logging tool, with a median absolute time difference significantly less than 1000 milliseconds.
- A content analysis of post-task questionnaires revealed that all themes related to important evidence and hypotheses were captured in the traces. Some themes related to domain knowledge or implicit assumptions were not directly captured but could be inferred.
- A trace-stimulated recall interview with one professional analyst confirmed that their self-explanation was consistent with the understanding derived from the trace analysis.
Limitations and Future Work
- Subtle Thoughts: The authors acknowledge that some subtle or quickly dismissed thoughts (implicit thoughts) might not be captured, potentially adding difficulty to interpretation.
- Trace Analysis Complexity: Analyzing the captured traces is itself a complex cognitive process, and further efforts are needed to develop guidelines or automated tools for trace analysis.
- Limited Recall Study: Only one participant's trace was fully analyzed via stimulated recall, which is insufficient to definitively conclude that trace-based understanding aligns with original cognitive processes. More recall studies are needed.
- Diversity of Strategies: The study observed a diversity of strategies used by participants for data exploration and hypothesis generation. Future work should focus on identifying, categorizing, and comparing these strategies to understand expert-novice differences and develop better training tools.
Summary updated