A temporal class activation map is a curve over time that shows which moments a classifier relied on when it labeled a whole signal window. For example, the classifier might say "a dishwasher ran somewhere in this 24-hour smart-meter trace." Thresholding that curve turns a cheap, window-level label into a timestamp-level answer: the dishwasher was on from 19:40 to 21:05. This trick is attractive in energy monitoring. There, window-level labels such as "this home owns a dishwasher" are nearly free, while second-by-second ground truth needs a separate meter on every appliance. The best-known result is striking. CamAL reports matching fully supervised localization on the IDEAL dishwasher data with 5,200× fewer labels.
This proposal asks whether such numbers survive two things practitioners will actually face. The first is that the threshold and other post-processing steps must be chosen without the timestamp labels the method is meant to avoid. The second is that the model must work in houses from a different dataset than the one it was trained on. We expect a substantial share of reported weakly supervised localization quality to come from two sources. One is post-processing tuned per dataset against ground-truth timestamps. The other is evaluation inside a single corpus. We propose a fixed, leave-one-dataset-out protocol across four public energy datasets to measure both effects. We also test whether cheap, label-free rules for choosing the threshold recover what oracle tuning provides.
Core Claim
Weakly supervised localization with activation maps has a hidden tuning step: the cutoff that turns a continuous map into on/off segments, along with smoothing and pooling choices. That step is the same kind of "validation trap" that PUBench documented for positive-unlabeled learning. In that setting, hyperparameters were chosen on validation data containing labels that would not exist in deployment. The research file notes that CamAL tunes its thresholding parameters per dataset. No 1D study compares activation-map variants under identical preprocessing and identical threshold rules.
We claim three things. First, when post-processing may only use source-dataset information, localization F1 on an unseen dataset drops measurably below the oracle-tuned figure. Second, the ranking of activation-map methods changes between the oracle and leakage-free protocols. Third, a de-biased map with a label-free threshold rule closes most of the remaining gap. The de-biased map adapts the fixes of Bae et al. to one-dimensional signals. The threshold rule is chosen only from weak labels and model self-consistency.
The three research directions play different roles here. Temporal activation mapping is the object under test. Benchmarking inconsistency supplies the hypothesis and the method: freeze every degree of freedom except the one being compared. Cross-dataset validation is the regime that makes the question matter.
Why This Is Not Already Done
The closest existing work is CamAL and its demo, DeviceScope. Both already evaluate across UK-DALE, REFIT, IDEAL and a private EDF dataset, and they motivate this proposal. They report relative gains over one weak baseline, CRNN Weak: 135% in F1 and 247% in Matching Ratio. However, each dataset is evaluated with its own threshold tuning, and no alternative activation-map mechanism is compared. We do not propose a new localizer to beat CamAL. We propose to determine how much of its advantage, and of any competitor's, holds under a frozen protocol.
Bae et al. diagnosed three biases in the standard pipeline, where features are averaged over the whole input (global average pooling) before classification:
- Averaging dilutes long events. Pooling over the whole window over-weights short spikes relative to sustained activity.
- Negative weights suppress true evidence. Negative classifier weights cancel real features that happen to resemble other classes.
- Max-normalization is fragile. Normalizing to the maximum value lets a single noise spike distort the map.
They fixed these with thresholded pooling, clamping negative weights to zero, and a percentile-based reference. Localization on CUB-200-2011 rose from 37.05% to 61.30%, but only for 2D images and only within one dataset. We supply the method here: we port those fixes to 1D and test whether they matter under dataset shift. Averaging dilution should be especially damaging for long, low-amplitude appliance cycles.
Wang et al. defined the 1D activation map for time-series classifiers but evaluated only classification accuracy on UCR. STPN introduced sparse attention alongside temporal activation maps for video. It also warned that such maps fire on background that co-occurs with the target, which is exactly the kind of cue we expect to break across datasets.
Our protocol design draws on three constraining results. Data Requirements for Applying Machine Learning to Energy Disaggregation found that supervised performance degrades below roughly 1–3 Hz sampling and that cross-house generalization needs tens of houses. It singled out UK-DALE and REDD as too small. That is a direct warning that single-corpus results on small datasets may not transfer. Revisiting Point Cloud Shape Classification showed that unifying training recipes can reverse architecture rankings, which motivates freezing the recipe here. EEG-FM-Bench standardized preprocessing across 14 EEG datasets but evaluated only sequence-level accuracy, not localization. It is a model for the kind of shared data pipeline we build.
Research Program
Datasets and task. We use four public corpora that share a washing machine label: UK-DALE, REFIT, IDEAL and ENERTALK. The three UK corpora also share a dishwasher label, so dishwasher experiments use those three.
- UK-DALE records sub-metered appliances every 6 seconds in five homes.
- REFIT covers 20 homes.
- IDEAL covers hundreds of homes, with survey data on which appliances each home owns.
- ENERTALK records 58 Korean homes at 10 Hz.
The sub-meters give timestamp ground truth that is used only for scoring. Models see only aggregate power and one binary label per window. In the main setting that label says whether the appliance ran in the window; in a harder setting it says only whether the home owns the appliance.
Frozen preprocessing. All data are resampled once to a common rate, cut into fixed-length windows, and scaled with per-house statistics. These choices are fixed before any model is trained. ENERTALK comes from a different country and grid, so we expect it to be the hardest transfer target; we report it separately.
Evaluation protocol. Evaluation is leave-one-dataset-out, with houses kept disjoint from one another. Every method gets the same architecture family, the same training budget, and the same number of hyperparameter trials. Everything is selected on source-dataset houses.
Threshold-selection regimes. Post-processing is chosen in four ways, applied identically to every method:
- Oracle-target: tuned on target-dataset timestamps. This is the leaky upper reference.
- Oracle-source: tuned on source-dataset timestamps, then frozen.
- Fixed default: a single percentile threshold set in advance.
- Proxy: chosen with no timestamp labels at all.
The proxy regime combines two signals that only need weak labels. The first is weak-label consistency: on held-out source windows, the thresholded mask should be non-empty exactly when the window label is positive. The second is agreement across the multi-kernel members of the ensemble: the chosen threshold should maximize overlap among masks from members with different kernel sizes. This mirrors how PUBench's proxy accuracy and proxy AUC replaced oracle labels for model selection.
Measurements. For each method we report three quantities:
- Oracle gap: oracle-target F1 minus proxy F1.
- Transfer gap: in-dataset F1 minus cross-dataset F1.
- Ranking change: Kendall's rank correlation between method rankings under the oracle-target and proxy regimes.
Localization quality is scored as timestamp-level F1 and as the Matching Ratio used by CamAL, so our numbers are directly comparable to theirs.
Baselines
Naive reference points. The do-nothing reference predicts "on" for every timestamp of a positive window and "off" elsewhere. This uses the weak label itself as the localization, so any method must beat it to show that the map adds timing information. A second naive reference thresholds the raw aggregate power change at a single cutoff chosen on the source data. This checks whether a learned model is needed at all for appliances with sharp power edges.
Strongest existing method. The strongest competitor is CamAL (Petralia et al.). It is an ensemble of 1D residual networks with kernel sizes 5, 7, 9 and 15. Their activation maps are combined into an attention mask and passed through a sigmoid to produce on/off output. It is the only 1D weakly supervised localizer in the file with a cross-dataset evaluation and a claim of parity with full supervision. On Lacuna we found only relative figures and no absolute localization F1 for CamAL on any single dataset:
- 135% F1 improvement and 247% Matching Ratio improvement over CRNN Weak, pooled over UK-DALE, REFIT and IDEAL.
- Parity with supervised models on the IDEAL dishwasher.
We therefore rerun CamAL ourselves on the same windows, with its ensemble structure intact, under all four threshold regimes. We first check that the oracle-target run reproduces a gain over CRNN Weak of roughly the reported size. We then use our own measured CamAL F1 as the bar.
Other activation-map methods. We run the following under the same budget:
- Single-network global-average-pooling maps from FCN and ResNet, as defined by Wang et al..
- Gradient-weighted maps in the style of Dutt et al..
- A 1D port of STPN's sparse class-agnostic attention.
- CRNN Weak, as CamAL describes it. No separate paper for CRNN Weak is in the file.
Supervised ceiling. A fully supervised sequence-to-sequence localizer trained on source timestamps sets the ceiling. It tells us how much of the cross-dataset drop is generic domain shift and how much is specific to weak supervision.
Ablations. The ablations isolate the two proposed ingredients. First, we add the 1D de-biasing fixes (clamping negative weights, thresholded pooling, and a percentile reference) one at a time to CamAL and to the single ResNet. Second, we swap the threshold regime while holding the model fixed.
What counts as success. The proposal's recipe is the de-biased map with the proxy threshold. It succeeds if, under leakage-free cross-dataset evaluation, it matches or exceeds rerun CamAL with oracle-source tuning. That comparison has to hold on at least three of the four held-out datasets, with confidence intervals from house-level bootstrap resampling that do not overlap.
Why not just…?
Why not just use a fixed percentile threshold and skip the tuning question? This is the strongest threat. Bae et al.'s percentile reference was designed precisely to be robust without tuning. If a fixed 90th-percentile cutoff comes within bootstrap error of oracle-source tuning on every held-out dataset, then the proxy rule adds nothing. We would report that directly. The benchmarking finding would then become "use a fixed percentile; oracle-tuned numbers overstate performance by the measured gap," which is still useful. The fixed-default regime is a first-class arm for exactly this reason. We expect it to fail mainly on long, low-amplitude cycles, where the right cutoff depends on how much of the window the appliance occupies. We report per-appliance results so this can be seen.
Why not just train a supervised localizer on one well-labeled dataset and deploy it everywhere? Sub-metered training data exist, but only for small numbers of houses. Supervised models need tens of houses to generalize across homes, according to Data Requirements for Applying Machine Learning to Energy Disaggregation. In contrast, possession labels in IDEAL-style surveys scale to hundreds of homes. Our supervised-ceiling arm measures whether supervised transfer from UK-DALE or REFIT beats weakly supervised transfer from IDEAL. If it does, the case for weak supervision in this domain weakens, and we would say so.
Why not just accept CamAL's own cross-dataset evaluation? CamAL tuned post-processing per dataset and compared against a single weak baseline. It therefore cannot separate how good the maps are from how good the tuning was. Our oracle-target versus proxy comparison on the same CamAL model answers this directly.
Why not just use EEG, where activation maps are already interpreted? The drowsiness models in ID3RSNet and Li et al. are validated qualitatively against alpha spindles and theta bursts. SADT, the dataset they use, has no timestamp-level event labels. Localization quality there cannot be scored, only admired. Energy data with sub-meters are the rare 1D setting where the ground truth exists.
Decisive Checks
Is the leakage problem real? The first check is whether the oracle gap is material. We will judge it material if proxy F1 falls several points below oracle-target F1 for most methods, and if rank correlation between regimes falls clearly below one. If both gaps are negligible, the leakage concern does not apply to 1D localization. That is a clean negative result, and it would be worth publishing as a validation of existing reports.
Does de-biasing help under shift? The second check is whether the 1D de-biasing fixes shrink the transfer gap more than they raise in-dataset F1. That pattern would show they remove dataset-specific cues rather than just sharpening maps. We will also inspect false-positive segments. The question is whether they concentrate on baseload fluctuations that co-occur with the appliance, the failure STPN described.
Does ENERTALK behave differently? The third check is whether ENERTALK, the foreign-grid, high-rate corpus, shows a qualitatively larger gap than the UK-to-UK transfers. A much larger gap would locate the remaining failure in the sensor and grid domain rather than in the choice of threshold.
References
- Few Labels are all you need: A Weakly Supervised Framework for Appliance Localization in Smart-Meter Series
- Accessible, Realistic, and Fair Evaluation of Positive-Unlabeled Learning Algorithms
- Rethinking Class Activation Mapping for Weakly Supervised Object Localization
- DeviceScope: An Interactive App to Detect and Localize Appliance Patterns in Electricity Consumption Time Series
- Time series classification from scratch with deep neural networks: A strong baseline
- Weakly Supervised Action Localization by Sparse Temporal Pooling Network
- Data Requirements for Applying Machine Learning to Energy Disaggregation
- Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline
- EEG-FM-Bench: A Comprehensive Benchmark for the Systematic Evaluation and Diagnostic Analyses of EEG Foundation Models
- The UK-DALE dataset, domestic appliance-level electricity demand and whole-house demand from five UK homes
- SleepXAI: An explainable deep learning approach for multi-class sleep stage identification
- ID3RSNet: cross-subject driver drowsiness detection from raw single-channel EEG with an interpretable residual shrinkage network
- A Compact and Interpretable Convolutional Neural Network for Cross-Subject Driver Drowsiness Detection from Single-Channel EEG