Mixing Pre-Training Data by Load Shape, Not Topic

Foundation models for sensor and telemetry data are usually pre-trained on whatever series are available, sampled uniformly. In energy data this means the model mostly sees the same thing over and over: flat baseloads, repeated weekday office curves, and many near-identical homes. The rarer patterns that forecasting actually fails on get very little weight. These include irregular households, bimodal residential peaks, and buildings that switch schedules. Language-model research has shown that clustering a corpus and then searching over how much of each cluster to include beats random sampling. That work clusters text by what it is about. We propose doing the same for a time-series foundation model, with one change: the clusters are defined by when and how consistently consumption happens, not by how much.

The core idea fits in two sentences. First, we split an energy pre-training corpus into behavioral archetypes. To do this we separate consumption volume from the normalized daily shape and add a measure of how regular each entity's routine is. Second, we use cheap small proxy models to search for the mixture of archetypes that minimizes forecasting error on a target population, and then pre-train the full model on that mixture. The claim is testable. At a fixed data and compute budget, archetype-weighted pre-training should beat uniform sampling, generic embedding clusters, and online robust reweighting on held-out load forecasting. The chosen weights should also say something interpretable about which behaviors the model was starved of.

Core Claim and Where It Sits

The method borrows its search procedure from Nemotron-CLIMB. That work embeds documents, clusters them with K-means, trains 62M–350M proxy models on sampled cluster mixtures, fits a predictor of downstream accuracy, and refines the weights iteratively. It reports a 5% domain-specific gain over random sampling and a 0.94 Spearman correlation between predicted and true accuracy. Its clusters are semantic and its data are static documents. We keep the proxy-search loop and replace the partition. The multilingual work on soft language clustering likewise reweights pre-training data by cluster, but it has no temporal representation.

Two results from temporal clustering motivate the choice of partition. SEGSys clusters consumers first by intensity tier and then by normalized daily profile, because otherwise large users dominate the distance metric and lump commercial and residential shapes together. Dent et al. discretize 5-minute smart-meter data with Symbolic Aggregate approXimation (SAX), a method that turns normalized segments into letter strings so that recurring shapes, called motifs, can be matched regardless of amplitude. They then cluster households by the variability of motif start times. This raised mean Corrected Rand Index across algorithms from 0.4549 to 0.5183 and isolated 26 "creatures of habit." Together these give a partition along three axes: volume, shape, and regularity. A semantic or generic embedding would not obviously expose these axes.

A third result shows that the partition matters for prediction at all. On the UNILAG campus data, Oyinlola and Oluseyi clustered 55 buildings with Mini-Batch K-Means and trained a separate forecaster per cluster. This cut MAPE by 33–36%. It also constrains the proposal, because in that study plain ARIMA beat LSTM and GRU: 7.2% MAPE campus-wide and 3.8–5.4% per cluster. We therefore cannot assume a neural model wins, and ARIMA stays in the comparison.

On the modeling side, we reuse the design of APEX, a decoder-only patched transformer pre-trained only on Wi-Fi telemetry. APEX beat the general model Toto by 18% MAE and SARIMA by 38% on proprietary data, but it trained on a single undifferentiated pool without asking which operating regimes dominate that pool. BehaveGPT approached the imbalance problem through Distributionally Robust Optimization (DRO), a loss that upweights whichever groups the model currently handles worst. It did this over event distributions, which makes it the natural online alternative to our offline mixture search. Mantis supplies a lightweight 2.2M-parameter encoder, which we use to build the competing embedding-based partition.

Research Program

The pre-training corpus is energy consumption series. It combines the energy-domain numerical histories in CAF-7M (7.4M windows drawn from 65 real datasets), downsampled aggregate channels from ENERTALK, and a training split of NESEMP households. Each entity-day is assigned to an archetype in three steps. We bin it into an intensity tier using total daily consumption. Within each tier, we cluster the normalized 24-hour profile with K-means, choosing the number of clusters by silhouette. Finally, we tag each entity with a regularity class from the SAX-motif start-time spread. The result is roughly 20 archetype cells, which matches CLIMB's cluster count so that the two search problems are equally hard.

The mixture search follows CLIMB. We train about 2M-parameter proxy transformers on Dirichlet-sampled archetype weights and measure each proxy's forecast loss on a validation split of the target population. We then fit a gradient-boosted predictor of loss from the weights, resample near the predicted optimum, and repeat for three rounds. The final model is roughly 10M parameters, APEX-Edge scale, pre-trained on the selected mixture. The search is run separately for each target population: campus buildings, UK households, and Korean households. This lets us check whether the selected weights differ in ways that make sense, for example more weight on irregular residential archetypes for NESEMP and more on sustained daytime blocks for campuses.

Baselines

The naive reference is the same 10M-parameter model pre-trained on uniform samples of the pooled corpus. To keep the comparison fair, it is given extra training tokens equal to the total compute spent on proxy search. Any gain from our method therefore cannot come from more compute.

The strongest existing mixture method is CLIMB's own recipe transplanted to time series. We cluster windows by their Mantis embeddings into the same number of clusters and run an identical proxy search with the same budget and predictor. This baseline is the direct test of whether temporal archetypes add anything over a generic learned partition. CLIMB's reported 5% gain over random sampling is a text result and no reported result found on Lacuna for embedding-clustered mixtures on time series. To beat it, the archetype mixture must achieve lower held-out MAE and CRPS on at least two of the three target populations, with non-overlapping confidence intervals across three seeds.

The strongest competitor on the actual forecasting targets is per-cluster ARIMA as in Oyinlola and Oluseyi, which reports 3.8–5.4% MAPE on UNILAG against 7.2% for whole-campus ARIMA. We rerun it on our splits with its reported protocol. Because the UNILAG loads are estimated from equipment inventories and are therefore smooth, we also run ARIMA on NESEMP, where 5-minute stochastic behavior is expected to hurt it. On UNILAG, matching the 3.8–5.4% range counts as parity, and improving on it counts as a win. General-model context comes from zero-shot and fine-tuned Chronos. No reported result found on Lacuna for Chronos on UNILAG, NESEMP, or ENERTALK, and no per-domain GIFT-Eval number is given in the file.

Two ablations isolate the key ingredients. The first is random partitions with the same cell sizes as the archetype partition, searched identically; it tests whether the archetype structure matters or whether any partition plus search helps. The second is archetype-balanced sampling with no proxy search; it tests whether the search itself earns its cost.

Why not just…?

Why not just cluster with a pre-trained embedding, as CLIMB does? This is the most serious threat. Mantis's multi-view tokenizer already includes instance-normalized signals and differences, so its embeddings may capture shape well enough. What the embedding probably does not separate is regularity: two households with identical average shape but different timing consistency. That is the axis on which Dent et al. found the most stable clusters. The embedding-mixture baseline settles the question directly. If it ties our method, we will report that the gain comes from the search and not the archetypes, and the regularity-only partition will show whether any of the temporal axes carries independent value.

Why not just use DRO over the archetypes, as BehaveGPT does over events? DRO avoids proxy models entirely and reweights online, which makes it cheaper. However, it optimizes the worst group rather than the target population, and it can overfit to noisy groups such as the unmetered "unknown" residual that ENERTALK documents. We run group-DRO over the same archetype cells at equal compute. If it matches the searched mixture on average error, the offline search is unnecessary, and we would recommend DRO.

Why not just cluster the targets and fit ARIMA per cluster? On smooth campus data this may well be the right answer, and the cited result shows neural models losing there. The foundation model's case rests on three things: new entities with little history, high-resolution stochastic household data, and a single model serving many sites. We therefore also report performance with 1% and 10% of target history, where per-entity ARIMA cannot be fit well.

Why not just fine-tune Chronos on the target data? General models transfer poorly to zero-inflated, bursty domain signals, as the APEX result against Toto shows. Still, fine-tuning gives a general model its best chance. It is run with the same target data and tuning budget as our model.

Decisive Checks

The proposal succeeds if the archetype-searched mixture beats all of the following at equal compute on held-out MAE and CRPS across UNILAG, NESEMP, and ENERTALK: uniform sampling, random-partition search, and embedding-cluster search. It also requires that the proxy predictor's rank correlation with full-model loss stays high, since CLIMB reports 0.94 in text and we need a comparably high value for the search to be trustworthy. Finally, the selected weights should shift across target populations in directions that match their known behavior.

The proposal fails informatively in two ways. If random partitions do as well as archetypes, the temporal clustering is decoration and the search alone explains any gain. If ARIMA per cluster still wins on NESEMP at full history, the domain may simply be too small for pre-training to matter.