Using Machine Learning in Adversarial Environments

Warren Davis, Daniel Dunlavy, Kevin Nauer, Matthew Letter, Karin M. Butler, Nicole Murchison, Chris Forsythe, Yevgeniy Vorobeychik2016 · DOI

TL;DR

This paper from Sandia National Laboratories argues that machine learning (ML) in cybersecurity fails because it is too predictable. By separating the act of prediction (what the ML thinks is happening) from the operational decision (how the defender actually responds), the researchers show how strategic deception and game theory can be used to "obfuscate" a system's defensive posture, making it significantly harder for both human and machine adversaries to find and exploit vulnerabilities.

What Problem Does This Paper Address?

In a typical supervised learning environment, we assume data follows a fixed distribution. In cybersecurity, this assumption is dead wrong. An intelligent adversary will deliberately change their behavior to evade detection once they understand the "rules" of the defender.

The authors identify three critical shortcomings in current ML-based defense:

  1. Determinism: Classifiers are usually deterministic, making them easy for attackers to reverse-engineer through probing.
  2. Conflation: Most systems treat a "malicious" prediction as an automatic trigger for action, ignoring operational costs and constraints (e.g., the cost of a false positive).
  3. Static Modeling: Standard ML doesn't account for the "Stackelberg" nature of the game—where the defender moves first by deploying a model, and the attacker responds by adapting their strategy.

Methods

The researchers propose a bi-level optimization framework that treats cyber defense as a game.

  • Separation of Concerns: They distinguish between the probabilistic classifier $p(x)$ (the ML's guess) and the operational decision function $q(x)$ (the actual action taken).
  • Stackelberg Games: They model the interaction as a game where the defender chooses a response strategy first, and the attacker observes and responds.
  • Strategic Obfuscation: Instead of a hard threshold (e.g., "block everything with a score > 0.9"), the defender can introduce randomized deception. For example, they might occasionally allow a known "malicious" event to pass to confuse the attacker about where the detection boundaries actually lie.
  • The Hybrid Toolkit: The team developed a simulation platform that allows for "plug-and-play" testing of both human and machine learning adversaries against these deceptive defenders.

Datasets

The authors initially attempted to use standard benchmarks but found them lacking for adversarial research:

  • KDD Cup 1999: Rejected because the "gaps" in the data were too obvious to be a realistic challenge.
  • SQL Injection Data: Generated using SQLMap, but rejected for human trials because the raw queries were too complex for non-experts to parse quickly.
  • Simple Function Data (Synthetic): The authors ultimately created a synthetic dataset based on a mathematical rule ($a < b * 2 + 1$) with a "nuisance" variable ($c$) to allow for spurious correlations. This provided a clean, learnable environment to test the impact of obfuscation.

Key Insights and Ideas

  • The "Aha!" of Obfuscation: The most striking finding was that obfuscation (lying to the attacker about whether their attack succeeded) actually slowed down machine learning adversaries. It forces the ML to see "noisy" labels, preventing it from finding a clean decision boundary.
  • Desirable Difficulties: In human trials, the group receiving 5% obfuscated feedback actually performed better than the control group. The authors suggest this might be due to "desirable difficulties"—a psychological concept where small amounts of noise or challenge force a learner to engage more deeply with the task, though they also acknowledge this could be a result of a small sample size.
  • Moving Target Defense: By treating the defense as a randomized strategy rather than a fixed rule, the defender creates a "moving target" that is much more expensive for an adversary to map out.

Key Results

  • Machine Learning Adversaries: Using a Logistic Regression adversary, the researchers found that obfuscation had a statistically significant impact on the learning rate ($p < .05$). When the defender "lied" about successes, the adversary's ability to reliably find holes in the defense dropped.
  • Human vs. Machine: As expected, the ML adversaries (Logistic Regression) performed better overall than humans, completing 100% of trials and achieving higher success rates. However, the study proved that the simulation framework could effectively measure both.
  • Optimal Obfuscation Rate: Preliminary data suggested an optimal obfuscation rate of roughly 6 per 100 trials. Too little doesn't confuse the attacker; too much might alert them that the system is deceptive.

Limitations and Future Work

The primary limitation is the use of synthetic data. While it allowed for controlled experiments, it lacks the messy complexity of real-world network traffic or malware.

Future directions suggested by the authors include:

  • Dynamic Obfuscation: Developing strategies that change the obfuscation rate in real-time based on the adversary's observed performance.
  • Algorithm Sensitivity: Testing how different ML architectures (e.g., Deep Learning vs. SVMs) handle deception. Some models with "flexible" boundaries (like Decision Trees) might be more susceptible to obfuscation than more "stable" models.
  • Attribution: Exploring whether the way an adversary responds to deception can be used to identify them (fingerprinting the attacker's learning style).

Summary updated

Abstract

Cyber defense is an asymmetric battle today. We need to understand better what options are available for providing defenders with possible advantages. Our project combines machine learning, optimization, and game theory to obscure our defensive posture from the information the adversaries are able to observe. The main conceptual contribution of this research is to separate the problem of prediction, for which machine learning is used, and the problem of computing optimal operational decisions based on such predictions, coup led with a model of adversarial response. This research includes modeling of the attacker and defender, formulation of useful optimization models for studying adversarial interactions, and user studies to meas ure the impact of the modeling approaches in re alistic settings.
BibTeX
@misc{davis2016using,
  author = {Warren Davis and Daniel Dunlavy and Kevin Nauer and Matthew Letter and Karin M. Butler and Nicole Murchison and Chris Forsythe and Yevgeniy Vorobeychik},
  title = {{Using Machine Learning in Adversarial Environments}},
  year = {2016},
  doi = {10.2172/1563076},
}
Figure