Testing Cyber Security with Simulated Humans

Jim Blythe, Aaron Botello, Joseph Sutton, David Mazzocco, Jerry Lin, Marc Spraragen, Michael ZydaIAAI 2011 · DOI

TL;DR

This paper introduces an agent-based system that simulates human behavior to test cybersecurity systems more realistically and cost-effectively. By modeling human errors, fatigue, and collaborative task performance, the system generates more nuanced network traffic and allows for the measurement of mission accomplishment under attack, rather than just penetration.

What Problem Does This Paper Address?

Traditional cybersecurity testing often relies on automated tools or limited human subject studies. Automated tools struggle to replicate the unpredictable nature of human error, which is a significant contributor to security breaches (around 60% according to a 2006 survey). Human subject studies, while more realistic, are expensive, difficult to repeat due to learning effects, and hard to scale for comprehensive testing. This leaves a gap in understanding how real-world human vulnerabilities and behaviors impact system security, especially in complex, collaborative environments. The authors aim to bridge this gap by creating a system that simulates human behavior in a repeatable and scalable manner.

Methods

The core of the paper is an agent-based simulation framework designed to mimic human behavior in network environments. Key methodological components include:

  • Belief-Desire-Intention (BDI) Architecture: Agents are built on a standard BDI model (SPARK framework), allowing them to maintain intentions, decompose tasks, and react to their environment. This provides a degree of flexibility beyond simple scripted behavior.
  • Realistic Software Interaction: Agents operate standard Windows software via a VNC-based controller (Skaion Corporation). This generates realistic network traffic and mimics how humans interact with systems.
  • Collaborative Task Performance: Agents are organized into teams to perform collaborative tasks, generating structured communication and traffic patterns that are more representative of real-world scenarios than typical statistical traffic generators.
  • Human Physiology and Fatigue Models: The system incorporates simple models of human fatigue, where performance degrades over time and agents may take breaks. Future work aims to include models of emotion and bounded rationality.
  • Reactive Planning: Agents use reactive planning to adapt their behavior and find alternative ways to achieve goals when unexpected events occur, such as resource unavailability.
  • Cyber Testing Infrastructure: A robust infrastructure supports hundreds of agents, each controlling a virtual desktop, managing process control, and facilitating inter-agent communication through a middleware system.

Datasets

Not discussed in the paper. The paper focuses on a simulated scenario and does not describe the use of pre-existing datasets for training or evaluation.

Key Insights and Ideas

  • Human error is a critical, yet hard-to-test, vulnerability: The paper highlights that human error is a major cause of security breaches, but its simulation in testing is challenging due to cost and repeatability issues.
  • Simulated humans offer a scalable and repeatable testing solution: The agent-based approach provides a way to introduce realistic human-like errors and behaviors into cybersecurity testing without the drawbacks of human subjects.
  • Mission accomplishment is a better security metric than penetration: By modeling agents with team goals, the system allows for measuring the impact of attacks on the overall mission success, which is often more relevant than simple penetration metrics.
  • Flexibility and reactive planning enhance resilience: Agents that can re-plan when faced with unexpected events (like disabled web services) demonstrate greater resilience to attacks, allowing for continued mission accomplishment.
  • Organizational structure significantly impacts emergent behavior: The paper shows how simple agent behaviors and their organizational roles (e.g., a bottleneck IT agent) can lead to complex, emergent network traffic patterns that are difficult to predict with traditional methods.
  • Fatigue and bottlenecks create bursty, unpredictable traffic: The simulation of an overloaded IT agent due to fatigue and a phishing attack demonstrates how human factors and organizational design can lead to periods of inaction followed by bursts of activity, creating a "bursty" traffic pattern.

Key Results

  • Increased Resilience with Redundancy: In an experiment where web sites providing information were disabled, agents with access to multiple redundant sites were significantly more successful in completing tasks than those with only one site available. For example, when 50% of sites were blocked, agents with three sites per cell achieved 9.5 goals on average, compared to 6 goals with only one site per cell.
  • Impact of IT Agent Fatigue: The simulation showed that the IT agent's rate of tiring had a disproportionate effect on team goal accomplishment. An IT agent that tired quickly and took frequent breaks led to a significant backlog of problems, drastically slowing down the team's progress. Conversely, a tireless IT agent or one with slower tiring rates allowed the team to achieve its goals more effectively.
  • Emergent Bursty Traffic Patterns: The scenario involving an insider attack and an overloaded IT agent demonstrated that relatively simple agent behaviors and organizational structures can lead to complex, emergent network traffic. This traffic pattern was characterized by periods of low activity followed by bursts of high activity, as the IT agent cleared backlogs and workers resumed tasks. This type of behavior is difficult to replicate with statistical traffic generators.

Limitations and Future Work

  • Current Limitations:
    • The current system includes simple models of fatigue but is developing models for emotion and bounded rationality.
  • Computer problems in experiments are simulated, and IT agents do not have real troubleshooting capabilities.
  • The current scenario is specific to an information-gathering task in a cloud-based spreadsheet.
  • Future Work:
    • Incorporating computational models of emotion based on appraisal and associative networks to better model stress and fear.
  • Integrating associative memory and appraisal with cognitive faculties like planning.
  • Developing models of bounded rationality to explain observed behaviors like risk-seeking.
  • Exploring the use of these agents in multi-agent behavioral simulations for scenarios beyond security testing.
  • Creating libraries of typical user profiles and organizational structures to facilitate rapid experiment instantiation.
  • Deploying the platform on a cyber testing range for more realistic testing.

Summary updated

BibTeX
@inproceedings{blythe2011testing,
  author = {Jim Blythe and Aaron Botello and Joseph Sutton and David Mazzocco and Jerry Lin and Marc Spraragen and Michael Zyda},
  title = {{Testing Cyber Security with Simulated Humans}},
  booktitle = {IAAI},
  year = {2011},
  doi = {10.1609/aaai.v25i2.18850},
}
Figure