Environment Replicas as Free Ensembles: Stacking the Snapshots of Communication-Efficient Pre-Training for Cross-Network Intrusion Detection

Network intrusion detectors look nearly perfect on the network they were trained on and then fail on a new one. A detector's recall can fall to 26% on a different topology. A device classifier can go from 0.997 to 0.592 F1 on an unseen network. Models latch onto local quirks: addresses, ports, and the particular shape of "normal" traffic at one site. Meanwhile, traffic data is naturally split across sites that cannot cheaply ship raw logs to one place. That makes communication-efficient training a practical necessity. In this kind of training, each machine trains on its own data for many steps and only occasionally syncs with the others. We propose to place each network environment on its own machine and run that infrequent-sync pre-training. Just before each sync, we save the per-environment model copies. We then learn a small combiner that decides how much to trust each environment's copy on traffic from a network none of them has seen. The combiner is trained only on cross-environment predictions.

The core observation is that the sync interval does two jobs at once. Longer intervals save bandwidth, and they also let each copy specialize to its environment. In ordinary communication-efficient training, that drift is a nuisance to be averaged away. In ensemble learning, it is the error diversity that a learned combiner needs. Our claim is that one setting of this knob buys both. At moderate sync intervals, the saved per-environment copies are diverse enough to beat their own average on unseen networks. They are also still close enough to be distilled or averaged afterwards, and the extra communication cost is zero.

Core Claim and What Is New

Three ideas are combined here. Stacked generalization, or stacking, trains a second-level model on held-out predictions of several base models so that it learns when to trust which one. Communication-efficient training in the DiLoCo style lets $M$ model copies take $H$ local steps between syncs. The method is characterized by Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo for dense text transformers, and it needs 10x to 100x less bandwidth while keeping compute utilization above 80%. Domain-specific self-supervised pre-training on network flows treats flow sequences like sentences, as in A Method for Network Intrusion Detection Using Flow Sequence and BERT Framework and NorBERT.

The new element is out-of-environment stacking over sync-interval snapshots. In ordinary stacking, the combiner is trained on out-of-fold predictions so that no base model scores data it was trained on. Here the folds are environments. For a labeled example $x$ from environment $e$, the combiner's input is built only from copies that never trained on $e$:

$$z(x) = \big[\, f_j(x) \,\big]_{j \neq e}, \qquad j \in \{1,\dots,M\},$$

where $f_j$ is the detector built on copy $j$'s saved encoder plus a small classification head trained on environment $j$'s labels. Copy $e$'s slot is filled with a learned "missing" token. On a truly new network every copy is out-of-environment, so the combiner faces the same situation at test time that it was trained on. This is the cross-environment analog of the leakage control that makes stacking work.

Closest Work and How This Differs

Three methods reduce communication by keeping copies mergeable and then collapsing them into one set of weights. WASH swaps a tiny fraction of parameters between identically initialized copies trained on the same data, then averages them. It cuts communication 200x on CIFAR, at about 82% versus 84% for an explicit ensemble on CIFAR-100. MERIT splits instruction-tuning data by gradient conflict, trains branches with no communication, and merges them. It beats joint training (54.3 to 57.0 on Qwen2.5-VL-3B), but only from a merge-ready base model. DiLoCo averages pseudo-gradients every $H$ steps.

All three throw away the per-copy specialization that we want to exploit. None partitions data by deployment environment. None evaluates on held-out environments. They supply our training machinery, and our proposal differs in what is kept and how it is combined.

Two stacking systems for intrusion detection motivate the gap from the other side. Effective network intrusion detection using stacking-based ensemble approach stacks KNN, SVM, and random forest under an XGBoost combiner on raw flow features with socket fields removed. It still drops to 78% accuracy from CIPMAIDS2023-1 to CICIDS2017. xIDS-EnsembleGuard stacks tree and recurrent models across datasets but never tests transfer to a held-out network. Both stack models trained on pooled or single-site data, so neither combiner ever sees what cross-environment failure looks like.

One paper constrains the design. Ensembling Tabular Foundation Models - A Diversity Ceiling And A Calibration Trap shows that similarly pre-trained models make nearly identical errors (pairwise Q-statistic 0.961). In that setting, stacking adds only +0.18% at 253x compute, and logistic-regression stacking gives the worst calibration. Our copies share an initialization and sync every $H$ steps, so they are exactly the kind of models that could hit this ceiling. We treat this as the central risk: the measured Q-statistic as a function of $H$ is a primary outcome, not an afterthought.

Research Program

We would build a small masked flow-sequence encoder of a few million parameters. It follows the Watabe and Nguyen recipe of masking flows within windows of adjacent NetFlow records, over standardized NetFlow v2 features with addresses and ports removed. The encoder is pre-trained with DiLoCo, one worker per environment. The environments are the NetFlow v2 versions of CIC-2018 and UNSW-NB15 used by DI-NIDS, ToN-IoT, CIDDS-001 internal and external, CIDDS-002, and CIPMAIDS2023-1, mapped to the common feature set where possible. Feature-set mismatches are a known engineering risk. Environments that cannot be mapped cleanly will be dropped rather than imputed.

We sweep $H \in \{1, 30, 100, 300, 1000\}$, plus a no-sync limit, at a fixed total number of flows processed. $H=1$ is equivalent to standard data parallelism. For each run we save each copy's inner weights just before the final sync. On each saved encoder, frozen, we train a light classification head on its own environment's labels. The combiner is deliberately simple: a regularized logistic model followed by temperature scaling fit on out-of-environment data. Its inputs are the out-of-environment copy scores plus their disagreement (variance across copies). The disagreement feature is meant to steer away from the calibration trap by letting the combiner become less confident when environment specialists disagree. This is a cheap stand-in for the density-based scoring that helped DI-NIDS. Bandwidth is accounted for exactly: we count bytes exchanged per run and report detection quality at matched bytes.

Evaluation is leave-one-environment-out throughout. One environment is held out entirely, from pre-training, from the heads, and from the combiner, and we cycle through all of them. We run two regimes. In zero-label transfer, no target labels are available. In few-label transfer, 3.7% of target labels are available, which matches the Always be Pre-Training setting, and those labels may only refit the combiner. Metrics are macro F1, attack recall, false-positive rate on benign traffic (the known weakness of masked flow models), expected calibration error, pairwise Q-statistic among copies, bytes communicated, and inference latency per 100 flows. Latency is reported so the comparison with the 43k-parameter FG-SAT at 4.82 ms is honest.

Baselines

The naive reference point is a single encoder of the same size and pre-training objective trained with ordinary synchronous data parallelism ($H=1$) on all training environments pooled, with one head on the pooled labels. This is "collect everything and train one model with identifiers stripped." We give it the same flows, the same compute, and the same hyperparameter search budget of a fixed number of trials per method. Beating it means higher held-out-environment macro F1 with no worse calibration.

The strongest existing cross-domain method in this benchmark family is DI-NIDS. It combines a domain-adversarial feature extractor, which uses a gradient reversal layer to remove environment-identifying information, with a one-class SVM. It reports an average cross-domain F1 of 89.54% with 6.42% degradation between NFv2-CIC-2018 and NFv2-UNSW-NB15, against 80–94% drops for conventional models. We will rerun it on our exact leave-one-environment-out splits with its published design and the same tuning budget. On the CIC-2018/UNSW-NB15 pair we will also check that our rerun lands near its reported 89.54%. Beating it means exceeding its average held-out F1 across all held-out environments, not just that pair. For the few-label regime, the reference is the self-supervised graph model of Always be Pre-Training, which reports 84.64% F1 on NF-UQ-NIDS-V2 with 3.7% labels. We rerun it on our splits rather than compare to that number directly. For raw-feature stacking we rerun the Ali et al. KNN/SVM/random-forest/XGBoost stack under leave-one-environment-out, where it reports 78% accuracy on its one cross-dataset transfer.

Two ablations isolate the key ingredient. The first keeps $H$ and the worker count fixed but assigns flows to workers at random rather than by environment. If stacking still helps, the gain comes from generic optimization noise rather than environment specialization. The second trains the combiner on in-environment predictions, so each copy scores its own data. We expect this leaked combiner to over-trust specialists and do worse on new networks; if it does not, out-of-environment construction is unnecessary.

Why not just…?

Why not just use the averaged DiLoCo model, fine-tuned? This is the most natural competitor, because averaging is what communication-efficient training was designed to produce. DiLoCo's own scaling results show averaged copies matching or beating data parallelism on text. Our proposal adds something only if the snapshot stack beats the averaged model on held-out environments. So the averaged model, given the same head training and target-label budget, is evaluated at every $H$. If the gap is near zero at all $H$, the claim is false. We would then report that environment drift in network pre-training is benign under averaging, which is itself a new and useful fact about DiLoCo outside dense text models.

Why not just average the copies' predictions uniformly, or pick them greedily? This is a real threat. Tanna et al. found that greedy selection matched cascade stacking with better calibration when base models were redundant. Uniform and greedy averaging over the same snapshots are run as baselines. A learned combiner is justified only if environments differ in which specialist transfers. For example, an IoT-trained copy might be more trustworthy on ToN-IoT-like traffic, and a uniform average cannot express that. If uniform averaging ties the combiner, we would drop the learned combiner and keep the cheaper rule. The proposal would then reduce to "snapshot ensembling," still with the $H$-diversity finding intact.

Why not just train fully independent per-environment models with no communication and stack them? Independent models maximize diversity at zero bandwidth, much like MERIT's communication-free branches. But each environment then pre-trains on far less data. The masked flow model has no shared base to start from, and the models cannot be averaged or distilled afterwards. MERIT itself notes that merging requires a shared basin. This is the no-sync endpoint of our $H$ sweep, so the experiment answers the question directly. We predict an interior optimum in $H$, but if no-sync wins we will say so.

Why not just align domains adversarially and skip ensembles? DI-NIDS shows that removing environment-identifying features works well on one environment pair. The two approaches are not exclusive: adversarial alignment discards environment-specific signal, while our combiner tries to route around it. We test DI-NIDS as a baseline and also as the head on each copy. If the combination gives nothing over DI-NIDS alone, stacking is redundant there.

Decisive Checks

The proposal stands or falls on two curves across the $H$ sweep at matched total flows. The first is the pairwise Q-statistic among saved copies. It should start near 1 at $H=1$ and fall as $H$ grows. If it stays near Tanna et al.'s 0.961 even at large $H$, copies are too similar to stack. The second is held-out-environment macro F1 of the out-of-environment stack minus that of the averaged model. It should rise from roughly zero at $H=1$, peak at an intermediate $H$, and flatten or fall in the no-sync limit as each copy's pre-training data shrinks.

A positive result means three things hold together. There is an intermediate $H$ where the stack beats the averaged model, uniform averaging, and DI-NIDS on held-out networks. Its calibration error is no worse than temperature-scaled single models. And the random-partition ablation shows a clearly smaller gain. In that case communication savings and ensemble diversity really are one knob, and environment-partitioned communication-efficient training is a principled way to build detectors that know when their training environment does not apply.