Large pretrained models for sensor data (time-series foundation models) almost all begin by rescaling each input window to zero mean and unit variance, a step called instance normalization. This makes training stable and lets one model handle signals measured in tons, watts or bytes. It also throws away the one fact that often matters most, which is how big the signal actually is. On oil-rig drilling data, a 10-ton jump in the weight hanging from the rig (the hookload) is routine at 4,000 m depth and catastrophic at 500 m. After instance normalization the two cases look identical. As a result, adapting Chronos-2 on 1.25 million hours of unlabeled rig data moved hookload anomaly F1 only from 0.51 to 0.52, while the same adaptation lifted scale-free torque anomalies from 0.47 to 0.69 (Are Time Series Foundation Models Ready for Oil and Gas Drilling Anomaly Detection?).
We propose to leave the pretrained model and its normalization untouched and to add back only the information normalization removed, expressed relative to what the operating context predicts. A small model, trained on unlabeled data alone, learns the typical level and spread of a window given static side information such as bit depth, house or appliance metadata, or network configuration. The gap between a window's actual statistics and this context-predicted reference, which we call the context residual, is fed into the frozen backbone as a feature-wise modulation. The modulation is then adapted to a new rig or house from a handful of labeled events. The bet is that "how unusual is this window's magnitude for this context" is the right scale signal. Raw magnitude does not transfer across contexts, and window-relative magnitude is erased by normalization.
Core Claim
Scale erasure in foundation models is best repaired inside the normalization step, by conditioning on side information, and not by dropping normalization or by concatenating raw statistics at the output. Concretely, let $x$ be an input window with its own mean $\mu_w$ and standard deviation $\sigma_w$, and let $z$ be the static side information for that window. A reference model with parameters $\phi$ outputs a context mean $m_\phi(z)$ and context spread $s_\phi(z)$. It is fit by Gaussian likelihood to predict $\mu_w$ and $\log \sigma_w$ on unlabeled windows. The context residual is
$$ r = \left(\frac{\mu_w - m_\phi(z)}{s_\phi(z)},\; \log \sigma_w - \log s_\phi(z)\right), $$
and each backbone layer's features $h$ are modulated as $h' = \gamma(r, z) \odot h + \beta(r, z)$. Here $\gamma$ and $\beta$ are small networks initialized to the identity. This is Feature-wise Linear Modulation (FiLM), a per-channel scale and shift.
Because the backbone still sees instance-normalized input, its pretrained features and optimization behavior are preserved. Because $r$ is measured against a context reference rather than against the window itself, it stays well defined for flat signals whose $\sigma_w$ is near zero. That is exactly the "Fridge Problem" that defeats window-level stationarization in NILMFormer.
The claim is falsifiable in a specific way. On magnitude-defined tasks such as hookload, the NILM appliance datasets and TelecomTS root cause, the method should close most of the gap left by the plain backbone. On shape-defined tasks such as torque, it should neither help nor hurt. Most of the gain should come from the context reference, not from the raw statistics alone.
How this differs from the closest work
Three papers each solve a piece of this problem.
NILMFormer motivates and partly supplies the method. It extracts each window's mean and standard deviation, feeds them back as a token, and projects outputs back to watts. However, it trains one supervised model on a fixed distribution, measures scale relative to the window rather than the context, and never adapts to new houses from few labels.
Task-Informed Meta-Learning (TIML) supplies the conditioning mechanism. A task encoder reads metadata such as coordinates and crop type and modulates hidden layers by FiLM inside a meta-learning loop. This raised CropHarvest mean F1 from 0.652 for standard MAML (model-agnostic meta-learning, which learns an initialization that adapts in a few gradient steps) to 0.802. TIML never touches normalization, however, and its metadata modulates features without restoring any quantity normalization removed.
TaskNorm constrains the design. It shows that normalization choices can make or break few-shot learning: transductive batch normalization falls from 98% to 22% on Omniglot when queries arrive one at a time. Our reference therefore must not depend on test-batch statistics, and in our design it depends only on the single window and its own side information. TaskNorm still maps everything to unit scale, treating magnitude as nuisance, while we treat it as signal.
Two further papers motivate the problem. The drilling study (Sengupta et al.) names scale preservation in foundation models as an open limitation. TelecomTS shows that an ad-hoc scale-aware branch on Autoformer adds over 30 percentage points to root-cause accuracy. Neither conditions scale on side information.
At the other extreme, Valovage shows raw-distance shapelets beat normalized ones, 81% against 42%, on electrical classification. That is evidence that magnitude matters, but it comes from nearest-neighbor methods, not deep pretrained models.
Label efficiency enters in two places. First, the reference model $m_\phi, s_\phi$ is fit without labels. Second, only the small FiLM networks and a head are adapted per deployment. This matches the lesson of the ECG benchmark (Benchmarking ECG FMs): pretrained backbones give 3.3x to 9x label efficiency, but struggle when the target depends on physical quantities the pretraining objective ignored.
Research Program
The work has three stages.
In the first stage we build the reference model. For each domain we define the side information $z$. For drilling it is bit depth, hole depth, pump pressure regime and rig identity. For NILM (non-intrusive load monitoring, which infers individual appliance usage from a household's total meter reading) it is house identity, time features and appliance category. For TelecomTS it is cell and configuration metadata. We fit $m_\phi, s_\phi$ as a small network on all unlabeled windows. We will inspect the result directly. For drilling, it should approximately recover the physically expected dependence of hookload on string length, and it should do so without being told the physics.
In the second stage we attach residual-conditioned FiLM to frozen backbones and adapt it. The backbones are Chronos-2 and Toto for drilling and telecom, and a pretrained NILM Transformer for load disaggregation. Tasks for adaptation are individual rigs or wells, houses, and network cells. We compare two adaptation rules on the same small modules. One is plain fine-tuning from a pooled pretrained head. The other is MAML-style meta-learning of the FiLM and head initialization across training rigs or houses, in the spirit of Meta-Adapters, which meta-learns small adapter layers on a frozen extractor.
We hold meta-learning to the same standard as every other component. It is kept only if it beats simple fine-tuning at 1 to 20 labeled events per new site.
In the third stage we analyze when context residuals help. We sort tasks by how much of the label signal can be predicted from window statistics alone. This is measured by a probe that sees only $(\mu_w, \sigma_w, z)$. We then test whether the method's gain tracks that quantity. This gives a practical diagnostic for when a user should expect scale re-injection to matter.
Baselines
The naive reference point is the frozen or domain-adapted backbone with standard instance normalization and a linear or small segmentation head. For hookload, the strongest reported number is Chronos-2 after self-supervised domain adaptation at 0.52 F1 (Sengupta et al.). For torque it is 0.69 F1 for the same model. For TelecomTS anomaly detection, Toto reaches 0.615 F1 (TelecomTS). Beating the naive point means a clear hookload gain, with 0.60 F1 as our target for a meaningful result, and no torque loss beyond seed noise.
The strongest scale-restoring competitor is NILMFormer-style statistics re-injection (NILMFormer). The window mean and standard deviation are projected into a token or into FiLM, on the same frozen backbone, with no context reference. On UKDALE and REFIT, NILMFormer reports a 15% MAE improvement and a 22% matching-ratio improvement over BERT4NILM and UNet-NILM. These are relative figures only; no absolute value appears on Lacuna, and we will reproduce the absolute numbers ourselves. This baseline receives identical data, backbone, label budget and hyperparameter search trials. The comparison against it is the key ablation, because the only difference is whether scale is measured against the window or against the side-information reference.
The strongest metadata-conditioning competitor is TIML (Task-Informed Meta-Learning). Here metadata $z$ alone drives FiLM, with meta-learning, but receives no residual. This isolates whether the benefit comes from knowing the context or from comparing the window to the context.
The strongest compact specialist is CNN-FNO, a 5M-parameter convolutional network combined with a Fourier neural operator, trained on raw signals with metadata channels. It reported 0.68 F1 on torque (Sengupta et al.). No reported result was found on Lacuna for CNN-FNO on hookload, so we will train it ourselves under the same label budgets. As a normalization-side baseline we add TaskNorm-I (TaskNorm) in the adaptation layers, where it matters for the meta-learned variant.
All methods share splits by rig or house, labeled-event budgets of 1, 5, 20 and all, five seeds, and equal tuning trials.
Why not just…?
Why not just stop normalizing? Raw-magnitude methods do win for nearest-neighbor shapelets (Valovage). However, the pretrained backbones were trained on normalized inputs. The drilling study reports that stripping their normalization breaks them, and raw magnitudes do not transfer across houses whose baseline consumption differs. We still include an un-normalized fine-tuned backbone and a raw-input CNN-FNO so the claim is tested, not assumed.
Why not just concatenate the window statistics and metadata into the classification head? This is the most serious threat. For hookload, a head that sees $\mu_w$, $\sigma_w$ and depth might learn the context comparison on its own. Our answer is empirical. We run this late-fusion baseline at every label budget. We expect it to match our method with many labels and to fall behind at 1 to 20 events, because it must learn the depth-to-weight relationship from labels, while our reference learns it from unlabeled windows. If late fusion ties at low budgets too, the lesson is that the unlabeled reference is what matters and FiLM is unnecessary, and we will report it that way.
Why not just normalize with a hand-built physical model, such as expected string weight from depth? Where such a model exists, it is a strong and cheap baseline, and we will include it for drilling. It does not exist for NILM appliances or telecom cells, so a learned reference is what makes the approach general. If the learned reference does no better than the hand-built one on drilling, that still validates the design as a way to recover physics-style normalization from unlabeled data.
Why not just fine-tune the whole foundation model on the labels? Full fine-tuning keeps instance normalization, so it cannot see the erased magnitude; the 0.51-to-0.52 hookload result shows adaptation alone does not fix this. It also costs the roughly 10x latency penalty of Chronos-2 over CNN baselines at every retraining. We include full fine-tuning at the matched label budgets.
Proposed Experiments and Decisive Checks
The primary benchmarks are the drilling hookload and torque segmentation sets. These contain about 167,000 and 94,000 windows, plus the 3,000-series unlabeled corpus. Access depends on the authors of Sengupta et al.. The public fallbacks are UKDALE and REFIT for NILM, with houses held out as new tasks, and TelecomTS (TelecomTS) for anomaly detection and root cause, with cells held out. As a secondary transfer test, we use PTB-XL (Deep Learning for ECG Analysis) with age and sex as $z$ on amplitude-dependent statements, where the CNN ensemble reaches 0.939 macro AUC on the full data. Metrics are per-second F1 for drilling, MAE and matching ratio for NILM, F1 and root-cause accuracy for TelecomTS, and macro AUC for ECG. All are reported as label-budget curves.
Four checks decide the project.
The first is a scale-swap test. We move a real hookload anomaly window to a synthetic context with a different depth $z$. Instance-normalized baselines are invariant by construction, so their predictions cannot change. Ours should flip in the physically correct direction.
The second is the torque control: gains must be near zero on scale-free anomalies. A gain there would suggest the method is only adding capacity.
The third is the flat-signal test. On NILM fridges and other steady loads, window-statistics re-injection should degrade while context residuals should not.
The fourth is the ablation chain. We compare metadata-only FiLM, statistics-only FiLM, and the full context residual, with and without meta-learning, at 5 labeled events. If the full residual does not beat both single-ingredient variants at low budgets, the core claim is wrong.
References
- Are Time Series Foundation Models Ready for Oil and Gas Drilling Anomaly Detection?
- NILMFormer: Non-Intrusive Load Monitoring that Accounts for Non-Stationarity
- Task-Informed Meta-Learning for Remote Sensing
- TaskNorm: Rethinking Batch Normalization for Meta-Learning
- TelecomTS: A Multi-Modal Observability Dataset for Time Series and Language Analysis
- Enhancing Machine Learning Classification for Electrical Time Series Applications
- Benchmarking ECG FMs: A Reality Check Across Clinical Tasks
- Meta-Adapters: Parameter Efficient Few-shot Fine-tuning through Meta-Learning
- Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XL