Hunting Log Incidents as Bumps in Time

Most log anomaly detectors learn what "normal" looks like once and then flag whatever deviates from it. That works on shuffled benchmarks and fails in production, where software updates, new workloads and runtime changes keep redefining normal. Particle physicists face the same problem when they look for a rare signal buried in a background that no simulation describes exactly. One answer they have found is to learn the background from the data on either side of the search region and compare it with what the region actually contains. We propose to carry this "sideband" method into log analysis, with time as the axis being searched. Each log session is mapped to a vector that captures its meaning. A conditional density model learns how these vectors are distributed in the hours before and after a candidate window, and its estimate is interpolated into that window. A classifier then separates the window's real sessions from sessions sampled out of the interpolated background. Its output estimates the ratio of the observed density to the expected one, and that ratio is high only where something new has appeared.

In plain terms, we turn a log stream into a search for a bump in time, the way a collider search looks for a bump in mass. The method needs no anomaly labels. It re-estimates normal behaviour locally, so slow drift is absorbed rather than flagged. Because it comes from physics, it brings a discipline log research mostly lacks: null "control channels" that show the detector is not inventing anomalies out of background fluctuations, and alarm rates stated per unit of traffic rather than as an F1 score.

Core Claim

The method we borrow is CATHODE, as applied in Isolating Unisolated Upsilons with Anomaly Detection in CMS Open Data. That work fits a normalizing flow (an invertible neural network that gives exact densities) to background in the mass sidebands, interpolates it into the signal window, and trains boosted decision trees to tell real window events from synthetic background. On CMS open data this raised a dimuon resonance from 1.6σ to 6.4σ, and a same-sign muon control channel confirmed that no fake peak had been carved out of the background.

Our claim has two parts. First, time, optionally together with a deployment or version tag, can play the role of the mass variable for log streams. Incidents are localized in time, while the background varies smoothly in time apart from known deploy events. Second, a semantic embedding supplies the continuous feature space the method needs. The CMS work operates only on continuous kinematic quantities and has no route to unstructured text; the embedding provides one.

Let $x$ be the embedding of a log session and $W$ a candidate time window. Let $p_{\rm data}(x \mid W)$ be the observed density inside $W$, and let $\hat p_{\rm bg}(x \mid W)$ be the flow's background density, fitted on sessions outside $W$ and conditioned on time. The classifier estimates

$$ R(x) = \frac{p_{\rm data}(x \mid W)}{\hat p_{\rm bg}(x \mid W)} = (1-f) + f\,\frac{p_{\rm s}(x)}{p_{\rm bg}(x)}, $$

where $f$ is the unknown fraction of incident sessions in $W$ and $p_{\rm s}$ is their density. The second equality holds when the background estimate is correct. Then $R$ ranks sessions in the same order as the true incident-to-background likelihood ratio, even though no incident was ever labelled. That is why the approach should hold up against the label noise that hurts supervised detectors.

The approach differs from the closest log work as follows. MPFM: Cross Multi-Domain Prototype Flow Matching for Log Anomaly Detection also uses a flow-style generative model. It learns one global model of normal sequences and scores sessions by how well that model reconstructs them and by their distance to learned prototypes. It never estimates a background that is local in time, and it never computes a density ratio against such a background.

Research Program

Representation. We embed each session (an HDFS block or a fixed time window on BGL, Thunderbird and Spirit) in two ways.

The first is the cheap embedding behind SemPCA. Following Try with Simpler - An Evaluation of Improved Principal Component Analysis in Log-based Anomaly Detection, we take GloVe word vectors, average them with TF-IDF weights into template vectors, and count templates per session. TF-IDF weighting down-weights words that appear everywhere.

The second is a Sentence-BERT embedding. Sentence-BERT is a transformer tuned to produce sentence vectors, and What information contributes to log-based anomaly detection? Insights from a configurable transformer-based approach identifies semantic information of this kind as the dominant signal for detection.

Both embeddings are reduced by PCA to 8 to 32 dimensions so that the conditional flow trains stably. The CMS work also operated in a few dimensions.

Background model and classifier. For each sliding window $W$, we fit a time-conditioned flow on the flanking sidebands. These are windows of equal width on both sides when we run the analysis retrospectively, and only on the past side when we run it causally. We then sample synthetic background into $W$ and train an ensemble of gradient-boosted trees to separate real from synthetic sessions in $W$, with k-fold cross-fitting so that no session is scored by a model it trained. Scanning $W$ across the stream gives a per-session score $R(x)$ and a per-window test statistic.

For the CAShift version and runtime shifts, the flow also takes the deployment tag as a condition. A known, planned change is then treated as part of the background. This plays the role that a known detector configuration change plays in physics.

Calibration and control channels. We report alarms per million sessions at fixed recall instead of F1 alone. This responds to the warning in Log-based Anomaly Detection with Deep Learning: How Far Are We? that F1 hides false-positive floods under heavy imbalance, such as Thunderbird's 0.5% anomaly rate.

The control channels play the role of the same-sign muon test. We run the full pipeline on stretches of each dataset that contain no labelled incident. We also run it on streams whose timestamps have been permuted within sessions of the same label: this spreads any real incident evenly across time, so no true time-bump should survive. In both cases the per-window statistic should follow its null distribution. Any excess is sculpting, meaning the method has manufactured a bump out of background.

Injection studies. We inject labelled anomalous sessions into clean windows at controlled fractions $f$. This maps sensitivity against $f$, which is the method's main known weakness: weakly supervised classifiers need enough signal in the window to learn from.

Baselines

Naive reference. The do-nothing reference is a per-session keyword-and-count rule that flags any session containing an ERROR- or FATAL-class template, run on identical chronological splits. It costs nothing and is the kind of hard cut that rose only to 1.6σ in the CMS setting. Beating it means higher recall at the same alarm rate on every dataset.

SemPCA. SemPCA from Try with Simpler - An Evaluation of Improved Principal Component Analysis in Log-based Anomaly Detection is the strongest cheap competitor. It reports an average F1 of 0.959 across five datasets against 0.983 for LogRobust, and it keeps roughly 0.91 F1 on Spirit with 1% of training data. We run it with the same embeddings, retrained on the same sidebands as our method, so the only difference is the scoring rule: linear subspace residual versus local density ratio.

LogFormer. The strongest supervised competitor is LogFormer: A Pre-train and Tuning Pipeline for Log Anomaly Detection, which pre-trains a transformer on one log source and adapts it to others with small adapter modules. It reports F1 of 0.98 on HDFS, 0.97 on BGL and 0.99 on Thunderbird under its original splits. Those numbers are not comparable to chronological splits; under chronological splits, DeepLog on BGL falls from 0.927 to 0.426 in Log-based Anomaly Detection with Deep Learning: How Far Are We?. We retrain LogFormer on chronological splits with its adapter budget and the same hyperparameter search budget, measured in GPU-hours, as our flow and classifier. We test it with and without 1% injected label noise. Our method should beat it on recall at a fixed alarm rate under chronological splitting and label noise. We do not expect to beat it on clean shuffled splits, and we will report if we do not.

MPFM. The strongest unsupervised generative competitor is MPFM. It reports an average F1 of 0.960 across HDFS, BGL, Thunderbird and Spirit under joint training. No standalone per-dataset number was found on Lacuna. We will run its released configuration under our chronological protocol.

CAShift detectors. On the shift benchmark, CAShift: Benchmarking Log-Based Cloud Attack Detection under Normality Shift reports an in-distribution average F1 of 0.90 across eight detectors, drops of up to 34% under shift, and up to 27% recovery with continual learning. Its BERT-initialized VAE with continual learning is the competitor to beat on post-shift F1 and post-shift alarm rate.

Ablations. Two ablations isolate the key ingredients. The first is direct sideband-versus-window classification without the interpolated flow. This is the simpler "classification without labels" variant: it trains the classifier on real sidebands instead of synthetic samples. The second removes the time conditioning from the flow.

Why not just…?

Why not just retrain SemPCA on a sliding window? That captures the "local background" part of the idea at almost no cost, and it is a real threat on HDFS, where anomalies are semantically blatant. But SemPCA scores by distance from a linear subspace. MPFM's analysis of anomalies that sit geometrically close to normal clusters suggests this rule misses incidents that only change the mix of otherwise normal templates. A density ratio is built to catch exactly those. The sliding-window SemPCA baseline uses identical windows, so if it matches us on recall at a fixed alarm rate, the density machinery has earned nothing and we will say so.

Why not just classify window against sidebands directly, without a flow? This is the most serious alternative, because it is simpler and also needs no labels. Its weakness is that anything that differs between sidebands and window becomes "signal", including ordinary drift and diurnal load. That is the analogue of mass-correlated features producing fake bumps, which is the problem interpolation is meant to fix. The ablation settles it. The direct classifier should show inflated null-channel excesses on drifting data such as the CAShift version shift. If it does not, we adopt the simpler method.

Why not just fine-tune a supervised detector like LogFormer? Supervised models depend on labels that are scarce and noisy in production. Under 1% label noise, LogRobust and CNN lose more than 36% F1 on HDFS (Log-based Anomaly Detection with Deep Learning: How Far Are We?). Our label-noise arm measures whether LogFormer's adapters inherit that fragility.

Why not threshold a flow's likelihood directly? Raw likelihood under one global flow is the MPFM-style one-class approach. It flags rare-but-normal sessions as readily as new ones, and it inherits every normality shift. The time-conditioning ablation, which amounts to an unconditioned global flow, measures this directly.

Proposed Experiments

All experiments use strictly chronological splits on HDFS, BGL, Thunderbird and Spirit, plus the application, version and runtime shifts of CAShift. The primary metric is recall at 1, 10 and 100 alarms per million sessions; F1 is secondary. We also report the null-channel false discovery rate and sensitivity as a function of injected fraction $f$.

Three outcomes decide the claim. First, on clean and timestamp-permuted control streams, our per-window statistic must stay within its nominal null rate while the direct-classification ablation exceeds it. Second, under the CAShift runtime shift to gVisor, the deploy-conditioned version must lose substantially less recall at a fixed alarm rate than the continual-learning VAE. Third, under 1% label noise, our label-free method must beat chronologically retrained LogFormer at the same alarm budget.

If the injection studies show that useful sensitivity requires $f$ far above the incident fraction actually found in real windows, we will narrow the claim to bursty incident classes rather than all anomalies.