Real-to-Sim Translation of System Logs for Fault Detectors Trained in Simulation

Labeled failures are the scarcest resource in log-based anomaly detection. Real clusters produce billions of log lines, real faults are rare, and labeling them by hand is impractical. Robotics and vision solve the same problem with simulators: inject the events you care about, record perfect labels, then close the gap to reality. For software logs, the simulator is easy to build. You can run a small containerized copy of the system and deliberately inject network drops, disk errors or crashed processes. What is missing is a way to close the gap between those clean simulated logs and messy production logs, which carry heartbeats, interleaved threads, timing jitter, cascades of duplicate alarms and shifted value distributions. This note proposes to close that gap the way Tactile Sim-to-Real Policy Transfer via Real-to-Sim Image Translation did for tactile sensors. Instead of making simulated data look real, we learn an unpaired translator (a model trained without matched before/after examples) that maps real log windows into the clean "simulator dialect." A detector trained entirely on labeled simulated faults can then run on production logs unchanged.

The idea fits logs especially well because of their structure. Each log line splits into a fixed template written by a developer and the runtime variables filled into it, such as IDs, latencies and return codes. Columnar log compressors already store logs in this form. We build the translator on top of that factorization, with one part that edits the template-ID sequence and one part that maps variable columns. That keeps translation cheap enough to run at ingestion rate, which is the binding constraint when data volume is the problem. The core claim is that most of the sim-to-real gap in logs is a lossy, many-to-one difference: reality adds noise that the simulator lacks. A one-directional, content-preserving translator from real to simulated logs should therefore recover fault-detection accuracy that a simulation-only detector loses. It should also do better than translating in the opposite direction or aligning hidden features.

What is new, and what is not

The closest work on logs is MPFM. It uses shared and private prototypes with flow matching to train one detector across four real systems (HDFS, BGL, Thunderbird, Spirit), reaching an average F1 of 0.960. MPFM transfers between real systems that each have their own labels. It never trains on simulated faults, and it adapts inside the detector rather than translating inputs. The closest work on method is PixelDA, which translates synthetic images into the real style. On synthetic-to-real LineMod it cut mean angle error to 23.5°, against 56.58° for domain-adversarial feature alignment (DANN, a method that trains hidden features to be indistinguishable across domains) and 53.27° for Domain Separation Networks. That result is the main reason we translate inputs rather than align features. It does not show which direction to translate, and logs are neither images nor continuous. The research file identifies sim-to-real transfer for system logs as unformulated (Gap 2), and we found no system in it that trains a log detector on injected simulated faults and deploys it on real telemetry.

Core Claim and Mechanism

Real logs contain more information than simulated ones. The file notes that under this kind of capacity asymmetry, cycle-consistency training (translate forward, translate back, require the original) pushes the generator to hide information in imperceptible patterns. The real-to-sim map is meant to throw information away: drop heartbeats, collapse duplicate cascades, undo thread interleaving, and normalize variable distributions. So we do not use a reverse generator. We use a one-directional translator $T$ that maps a real window $r$ to $T(r)$. Training uses an adversarial term against simulated normal windows and a contrastive content term in the style of Contrastive Unpaired Translation. The contrastive term asks each surviving position in $T(r)$ to stay identifiable with its source position in $r$. One further term encodes the asymmetry: $T$ may delete or merge events but may not invent template IDs absent from the input.

$$\mathcal{L}(T)=\mathcal{L}_{adv}(T(r),s)+\lambda\,\mathcal{L}_{NCE}(T(r),r)+\mu\,\mathcal{L}_{ins}(T(r),r)$$

In this objective, $s$ is a simulated normal window. $\mathcal{L}_{adv}$ is the discriminator loss, $\mathcal{L}_{NCE}$ is the patchwise contrastive loss computed over aligned template and variable positions, and $\mathcal{L}_{ins}$ penalizes probability mass placed on template IDs not present in $r$. $\lambda$ and $\mu$ are weights. The template part of $T$ is a small edit model over template-ID sequences, a lightweight relative of the hierarchical encoder in ContraLog. The variable part maps each variable column under the same template conditioning.

Only normal data is used to train $T$, on both sides. This means $T$ never needs labeled real faults. It also means a real fault should pass through as "something the simulator would also emit under that fault," because content preservation forbids deleting events that have no simulated normal counterpart. That is the hypothesis most likely to fail, and we test it directly.

Decoupled training plays a secondary role, as a deployment constraint rather than the core. Production logs drift, so $T$ must keep adapting on a live stream. We will test whether $T$ can be updated with simultaneous blockwise local objectives, as in Blockwise Self-Supervised Learning at Scale. In that setup each block discards its activations immediately, and the method came within 1.1 points of end-to-end backpropagation on ImageNet (70.48% vs 71.57%). We keep this only if it buys real throughput. Synthetic gradient modules (Static SGMs) are a weaker candidate, because their accuracy degrades when more than about three layers are updated under distribution shift, which is exactly our setting.

Research Program

We first build the benchmark the field lacks. The main source domain is a containerized Hadoop mini-cluster running MapReduce jobs similar to those behind the real HDFS dataset. Using the same software keeps the template vocabulary largely shared, so the gap is mostly noise, timing and volume. A fault-injection harness produces block-level labels for several fault types, such as datanode crash, network partition, disk write failure and slow node. The target domain is the public HDFS dataset: 11,175,629 records from more than 200 EC2 nodes, labeled at the block level.

A second, controlled-gap source lets us vary the gap deliberately. We take real HDFS normal sessions and "clean" them in known ways: remove heartbeats, deinterleave threads, quantize timing, and clip variable ranges. This yields a pseudo-simulator where we know exactly which corruptions separate source from target. We also report cross-template transfer to BGL and Thunderbird only as a stress test. We expect failure there and say so in advance, since the file notes pixel-level methods do not bridge structural gaps either.

Translation runs on the columnar representation produced by the parser. We cache per-template computation, which ContraLog reports removes 89.1% of encoder forward steps on Thunderbird. We report lines per second alongside accuracy.

Baselines

The do-nothing reference trains the detector on labeled simulated faults and applies it directly to real HDFS windows. The gap between this number and the upper bound defines what there is to recover.

The strongest simple detector is a Random Forest on parsed template-count features. We include it because A comprehensive study of machine learning techniques for log-based anomaly detection found that Random Forests and SVMs match or beat LSTM, DeepLog, LogRobust and NeuralLog across seven datasets, with the least hyperparameter sensitivity. We use it as the downstream detector in every arm, so gains come from the input transformation rather than the classifier. A GR-BiLSTM detector in the style of BERT-LogAnom serves as a second detector to check robustness.

The strongest adaptation competitor is DANN-style feature alignment, which aligns the detector's hidden features between simulated and unlabeled real windows. It gets the same unlabeled real normal data, the same detector backbone and the same tuning budget, a fixed number of hyperparameter trials selected on a small held-out real-normal reconstruction criterion that uses no real fault labels.

The opposite-direction competitor is PixelDA transplanted to logs. It translates simulated windows into the real style, including injected noise, and trains the detector on the translated labeled data. Its generator matches $T$ in capacity.

For upper-bound reference only, we cite in-domain results on real HDFS with real labels. BERT-LogAnom reports F1 89.98% on the standard split and 90.04% on a temporal split with dynamic thresholding. ContraLog reports F1 83.12%. MPFM reports 0.960 average F1 across four systems. No reported result was found on Lacuna for any sim-to-real log transfer setting, so there is no published baseline number for the actual task.

Beating the field means three things. First, real-to-sim translation recovers at least half of the F1 gap between do-nothing and our own supervised-on-real Random Forest. Second, it does significantly better than both DANN and sim-to-real translation across five seeds. Third, it matches or exceeds an unsupervised detector trained on real normal logs on fault types the real training data never shows.

Three ablations isolate the key ingredients. Replacing the one-directional objective with cycle consistency tests the capacity-asymmetry argument. Removing $\mathcal{L}_{ins}$ tests whether hallucinated events cause false alarms. Collapsing the template/variable factorization into a single token-level model tests whether factorization costs accuracy, or only buys throughput.

Why not just…?

Why not just hand-write a canonicalizer that strips heartbeats and duplicates, then apply the simulation-trained detector? This is the most serious threat. Tools like LogCleaner already filter repetitive "anti-events." The controlled-gap experiment settles it directly: the hand canonicalizer can be built to undo exactly the corruptions we introduced, so on that source it is an oracle. If $T$ does not get close to the canonicalizer there, the idea fails. On the real Hadoop mini-cluster, where the gap is unknown, $T$ must beat the best canonicalizer we can write in a fixed engineering budget. If it does not, we will report that hand rules suffice for same-software transfer and narrow the claim to drifting or unfamiliar deployments.

Why not just align hidden features with DANN instead of translating inputs? PixelDA's LineMod and MNIST-M results (83.2% for feature-level baselines vs 98.2% for translation) favor input translation, but that evidence comes from images. Log windows are short discrete sequences, and the case may be different. DANN is therefore a full baseline under identical budgets, not a dismissed alternative.

Why not just translate simulated logs into realistic ones, as PixelDA does? Generating realistic noise requires modeling the extra entropy of real logs, which the generator must invent. Deleting that noise is the easier direction. Real-to-sim also leaves the detector fixed and moves all adaptation into a cheap preprocessor. That said, sim-to-real translation yields labeled real-looking data that any detector can use, so the direct comparison decides between them.

Why not just skip simulation and train an unsupervised detector on real normal logs? That approach needs no translator and works well on frequent anomaly patterns. Simulation is useful only if it teaches the detector fault types it has not seen in real data. We therefore evaluate per fault type and hold out from any real-data tuning the fault classes that appear only through injection. If the unsupervised detector matches us there, simulation adds nothing.

Why not just retrain the translator with ordinary backpropagation every night? For a small factored translator this may well be cheap enough. Blockwise local training stays in the proposal only if it delivers measurable gains in throughput or freshness under drift. Otherwise we drop it.

Decisive Checks

The first decisive check is the controlled-gap experiment. Because we know the true corruptions there, we can measure whether $T$ removes them, by template-frequency divergence to the clean source, and whether detector F1 tracks that removal. The second is fault pass-through. On real HDFS anomalous blocks, we count how often $T$ deletes the events the detector needs, against how often it invents events in normal blocks. The cycle-consistency and no-$\mathcal{L}_{ins}$ ablations should show opposite failure modes here. The third is throughput: the full pipeline must sustain HDFS-scale ingestion, reported as lines per second on one GPU and on CPU. BERT-LogAnom reports 0.9 s per log on Thunderbird, so a per-line cost of that order would disqualify the approach for high-volume streams. The fourth is drift. Using a temporal split of HDFS, we compare a translator adapted online with simultaneous blockwise local losses against nightly backpropagation retraining, at equal compute, measuring F1 decay over time.