A likelihood ratio anomaly detector for identifying within-perimeter computer network attacks

Justin Grana, David H. Wolpert, Joshua Neil, Dongping Xie, Tanmoy Bhattacharya, Russell BentJ. Netw. Comput. Appl 2016 · DOI

Abstract

As cyber threats evolve, the "hard shell, soft center" model of network security has become increasingly vulnerable. Once an attacker bypasses perimeter defenses, they can often move laterally with minimal detection. Traditional anomaly detection systems frequently suffer from high false-positive rates due to the inherent unpredictability of legitimate user behavior. This paper proposes a novel Likelihood Ratio Detector that explicitly models attacker behavior as a stochastic process. By evaluating the probability of observed network traffic under both benign and malicious hypotheses, the detector achieves superior performance, particularly in the low false-positive regimes critical for enterprise security operations.

1. Introduction

The modern enterprise network is a complex ecosystem where the traditional focus on perimeter security—firewalls and signature-based intrusion detection—is no longer sufficient. Attackers frequently gain entry through social engineering or unpatched vulnerabilities, after which they seek to move laterally from their initial point of compromise to high-value targets.

To counter this, security teams often deploy anomaly detectors. These systems flag any activity that deviates significantly from a learned baseline of "normal" behavior. However, human activity is naturally "noisy." A developer pushing a large code update at 3:00 AM might be unusual, but it is not necessarily malicious. In large-scale environments, these benign anomalies trigger a flood of false positives, leading to "alert fatigue" and the eventual dismissal of genuine threats.

2. The Likelihood Ratio Framework

The core contribution of this research is the shift from simple anomaly detection to a likelihood ratio approach. A standard anomaly detector asks: How likely is this data given normal behavior ($P(D | H_0)$)? If the probability is low, an alarm is raised.

In contrast, the Likelihood Ratio Detector asks: Is this data more consistent with normal behavior or with an active attack ($P(D | H_1)$)? The decision is based on the ratio: $$\Lambda(D) = \frac{P(D | H_1)}{P(D | H_0)}$$ By incorporating a specific model of how an attacker behaves ($H_1$), the system can differentiate between "weird but benign" noise and "weird and malicious" patterns.

3. Modeling Stochastic Lateral Movement

To implement this framework, the authors developed a mathematical model of lateral movement within a network. They treat network traffic as a series of events governed by Poisson processes.

3.1 Normal and Malicious Traffic

Under the null hypothesis ($H_0$), nodes exchange messages at a calibrated "benign" rate. Under the alternative hypothesis ($H_1$), a compromised node begins sending additional "malicious" messages to its neighbors.

3.2 Infection Logic

The model assumes a simple but effective infection mechanism: if a compromised node sends a malicious message to a clean node, the target node becomes compromised with a probability of 1.0. This creates a cascading effect that mimics the spread of an attacker through a network as they scan for credentials or sensitive data.

4. Computational Challenges and Monte Carlo Integration

Calculating the exact likelihood of an attack is computationally intensive. Because the exact time a node was compromised is often unknown, the detector must account for all possible infection paths and timings. This involves solving high-dimensional integrals that are analytically intractable.

To overcome this, the researchers employed Monte Carlo methods, specifically a variant of the Gillespie algorithm. By simulating thousands of possible infection trajectories, the system can approximate the true likelihood ratio. While computationally demanding, this approach allows for a more nuanced detection strategy than simple threshold-based systems.

5. Experimental Evaluation

The detector was tested against two distinct datasets to validate its efficacy:

  1. Stylized Topologies: The researchers used controlled graph structures (e.g., star and caterpillar formations) to observe how the detector handled different attacker strategies, such as slow-and-stealthy versus fast-and-aggressive movements.
  2. LANL Authentication Logs: To test real-world applicability, the authors used nine months of authentication data from Los Alamos National Lab. This dataset, featuring over 11,000 users and 22,000 computers, provided a realistic background of "normal" noise.

5.1 Key Results

The primary metric for success was the Receiver Operating Characteristic (ROC) curve. The Likelihood Ratio Detector consistently outperformed baseline anomaly detectors across various network configurations. Notably, the performance gains were most significant at low false-positive rates. In a real-world security operations center (SOC), this translates to fewer "junk" alerts and a higher confidence in the alarms that are raised.

5.2 Robustness to Misspecification

A critical finding was the detector's robustness. Even when the researchers provided the detector with incorrect parameters regarding the attacker's speed or pathing strategy, it still outperformed the anomaly-only baseline. This suggests that having even a generalized model of "malicious intent" is significantly more powerful than having no model at all.

6. Conclusion and Future Directions

The Likelihood Ratio Detector represents a significant step forward in identifying internal network threats. By modeling the "why" behind the data, it provides a more surgical tool for security analysts.

However, several challenges remain for production-scale deployment. The current reliance on Monte Carlo integration is computationally expensive, suggesting a need for more efficient sampling or mathematical approximations. Furthermore, future iterations of the model should account for non-stationary "normal" traffic (which varies by time of day) and the possibility of multiple, non-colluding attackers. The authors also point toward game-theoretic models where the detector and attacker dynamically adapt their strategies in a competitive environment.

Summary updated

BibTeX
@article{grana2016likelihood,
  author = {Justin Grana and David H. Wolpert and Joshua Neil and Dongping Xie and Tanmoy Bhattacharya and Russell Bent},
  title = {{A likelihood ratio anomaly detector for identifying within-perimeter computer network attacks}},
  journal = {J. Netw. Comput. Appl.},
  year = {2016},
  doi = {10.1016/j.jnca.2016.03.008},
}

Loading related content...

Figure