To detect anomalies in a changing world, machine learning models must be dynamic. If a network's traffic patterns shift due to a new product launch, or a mobile robot transitions from day to night (Wulfmeier et al.), a static model will trigger a cascade of false alarms. This necessity drives the adoption of online learning, streaming analytics, and continuous retraining.
However, this adaptability introduces a fundamental paradox: if a system continuously updates its definition of "normal" based on incoming data, an adversary can manipulate that definition. This vulnerability is known as baseline poisoning or training phase contamination (Carle et al.). By injecting carefully crafted inputs, attackers turn a model's learning mechanism against itself, gradually expanding the boundaries of what the system considers benign until malicious activity can pass through undetected.
The Two Windows of Vulnerability
Defensive models establish their baselines through two primary operational phases, both of which present distinct targets for an attacker: initialization and continuous adaptation.
1. Initialization Contamination (The Cold Start Problem)
Many unsupervised anomaly detection frameworks operate under the assumption that their initial training data represents a clean, uncompromised baseline of normal operations (Karimipour et al.; Xiong et al.). If an adversary is already active when the system begins profiling, the model bakes the malicious behavior directly into its foundational baseline.
As Shao notes, if a botnet is already present during the initial traffic-profiling phase, the detector will fail to recognize subsequent botnet traffic as anomalous. This vulnerability spans multiple domains:
- Internet of Things (IoT): Meidan et al. demonstrate that if an IoT device is compromised prior to deployment, behavioral anomaly detectors calibrate themselves to accept botnet traffic as normal.
- Container Security: Wan et al. show that mining security sandboxes while a system is infected permanently encodes malware actions as authorized system calls.
- Cyber-Physical Systems: Cyber-attacks occurring during startup can force controllers to adopt incorrect physical models, blinding them to subsequent sensor spoofing (Lee et al.; Mo et al.).
To mitigate this, systems like Kitsune (Mirsky et al.) must rely on secondary, signature-based filters during their initialization periods to guarantee a clean baseline.
2. The Slow-Burn Shift (Continuous Retraining)
For systems that survive initialization cleanly, the threat shifts to the online update phase. Here, attackers exploit the model's need to accommodate concept drift—the natural evolution of data streams over time (Bhatia et al.).
Instead of launching a sudden, high-impact attack that would immediately trigger an alert, the adversary executes a "slow-burn" manipulation (Ongun et al.). By injecting subtle, near-normal data points over time, the attacker incrementally drags the model's decision boundary toward the target anomalous state (Kloft et al.; Kravchik et al.).
[ Pristine Baseline ] ---> [ Attacker Injects Near-Normal Samples ] ---> [ Shifted Baseline ]
| |
(Flags Attack) (Accepts Attack)
Why Models Fall for the Trap
To understand why online models are so susceptible, we must look at how they optimize their parameters. When an online anomaly detector (such as a streaming autoencoder or an ensemble) processes a new batch of data, it faces a fundamental dilemma: is this data point a transient outlier, or is it the beginning of a new normal?
As Sankararaman et al. point out, if a model updates its parameters too aggressively to minimize false positives during distribution drift, it inadvertently redefines malicious outliers as legitimate states. Attackers exploit this by structuring their inputs to mimic natural drift.
The optimization objective itself—minimizing reconstruction error or maximizing likelihood over a sliding window—becomes the vector of manipulation (Yang et al.). In systems like learned database components, this incremental adaptation allows malicious actors to systematically degrade model accuracy while remaining invisible to standard threshold-based detectors (Zhang et al.).
Designing Resilient Baselines
Defending against baseline poisoning requires breaking the feedback loop between unverified incoming data and model parameter updates. Several core strategies have emerged to decouple learning from contamination.
Isolation and Pre-Filtering
Instead of blindly incorporating every new observation into the historical baseline, systems can implement strict pre-filtering. Bhatia et al. demonstrate that by preventing highly suspicious microclusters from influencing the model's historical memory, the system maintains its sensitivity and prevents long-duration attacks from shifting the baseline. Similarly, ensembles of diverse unsupervised models can filter out contaminated training data by requiring consensus before an entry is allowed to alter the baseline (Xi et al.).
Analyzing Learning Dynamics
Rather than analyzing data points in isolation, defenses can look at how the model learns over time. Alex et al. propose using a small, verified set of clean data to establish a trusted anchor for expected learning trajectories. Unverified samples that cause anomalous deviations in loss trajectories can then be isolated and discarded.
In semi-supervised settings, Carlini demonstrates that poisoned data points often exhibit high mutual interdependence across training epochs compared to natural data. By monitoring the stability of predictions for unlabeled examples relative to their neighbors, defensive algorithms can identify and isolate coordinated poisoning attempts before they corrupt the global model.
Key Foundational Work
Network attack detection and defense - Manifesto of the Dagstuhl Perspective Workshop, March 2nd-6th, 2008 (2009) — contributed 1 concept elements to this direction.
Online Anomaly Detection under Adversarial Impact (2010) — contributed 1 concept elements to this direction.