A Physics-Shaped Bottleneck for Seeing Rooftop Solar Behind the Meter

Utilities increasingly cannot see what is happening at the edge of their own networks. A smart meter records only net load, which is household consumption minus whatever the rooftop panels produce. A feeder that shows 5 kW may actually be carrying 50 kW of demand offset by 45 kW of solar, and a passing cloud can erase that offset in seconds. Methods that separate net load into its hidden demand and solar parts, called disaggregation, now reach useful accuracy with deep networks. They give an operator no account of why they inferred a given solar output, though, and no way to correct them when the operator knows better, for example when an inverter has tripped offline. The interpretable-ML community offers "concept bottlenecks", models forced to predict human-meaningful intermediate quantities before the final answer. Those need expensive concept labels, and additive variants cannot represent the multiplication at the heart of solar physics.

We propose a disaggregation model in which the solar estimate is, by construction, the product of a few physically named quantities. Three are fixed per home: effective capacity, panel tilt and azimuth. One changes over time: a local attenuation factor between 0 and 1 that absorbs clouds, shading and inverter availability. Neural networks predict only those quantities, from the net-load history and irradiance data, and a fixed physical conversion formula turns them into kilowatts. Because the formula pins down what each quantity means, no concept labels are needed. Because nothing reaches the output except through these quantities, the explanation is faithful by construction, and an operator can override any of them at test time. The central bet is that this structure costs little accuracy when data are plentiful, and that it wins clearly in the regimes grid operators care most about: scarce data, meters that clamp exports to zero, and deployment in new regions.

Core Claim

The model predicts gross solar output $\hat s_{h,t}$ for home $h$ at time $t$ as

$$\hat s_{h,t} = C_h \, a_{h,t} \, \Pi(\beta_h, \gamma_h; G_t, D_t, N_t, T_t),$$

Here $C_h$ is effective capacity, $\beta_h$ and $\gamma_h$ are tilt and azimuth, and $a_{h,t}\in[0,1]$ is local attenuation. $\Pi$ is the normalized plane-of-array output of the PV Watts conversion model, computed from global horizontal, diffuse and direct normal irradiance $G_t, D_t, N_t$ and ambient temperature $T_t$. Demand then follows as $\hat d_{h,t} = P^{\text{net}}_{h,t} + \hat s_{h,t}$. The static quantities come from a small per-home encoder over long net-load history. The attenuation comes from a temporal network that reads a short window of net load alongside weather. Its job is to spot the sharp, weather-coherent jumps in net load that ordinary household demand does not produce.

The claim has three parts. On the Ausgrid Solar Home data, the model comes within a small margin of black-box Transformers at full data. It degrades less than they do when training data are cut to a cold-start fraction and when meters are censored at zero export. Overriding its concepts with operator-side information, such as registry capacity or attenuation measured at nearby sub-metered homes, reduces error substantially without retraining.

Relation to Prior Work

Two papers by Chen et al. motivate this work and serve as its accuracy targets. One is a personalized federated Transformer that treats net load and irradiance components as tokens (Privacy-Preserving Personalized Federated Learning for Distributed Photovoltaic Disaggregation Under Statistical Heterogeneity). The other is a multi-scale pooling-plus-attention network (Season-Independent PV Disaggregation Using Multi-Scale Net Load Temporal Feature Extraction and Weather Factor Fusion). Both argue that the interaction between load rhythms and irradiance needs high capacity. Both leave the latent space unconstrained, so no internal quantity can be inspected or edited. We keep neural capacity but spend it only on predicting physical quantities.

Two papers supply the physics we reuse. The remote-sensing pipeline of Remote-Sensing-Based Estimation of Rooftop Photovoltaic Power Production Using Physical Conversion Models and Weather Data shows that PV Watts, fed with detected panel attributes and weather reanalysis, estimates unmetered generation well. It never touches meter data, however, and it inherits the brittleness of aerial detection across imagery sources that is documented in A crowdsourced dataset of aerial images with annotated solar photovoltaic arrays and installation metadata. We move the conversion model into a learned disaggregator and use registry metadata only as an optional intervention.

Three interpretability papers supply the lens and also constrain the design. Concept Bottleneck Models introduced predicting labeled concepts first and editing them at test time. On knee radiographs, a CBM trained on a quarter of the data matched a standard model trained on all of it, and intervening on two concepts cut error from above 0.4 to about 0.3. Our model differs in that the physics equation replaces concept labels. Neural Additive Models gives exact per-feature shape plots but forbids interactions. It therefore cannot express capacity times irradiance times orientation, which is why we use a multiplicative structure rather than an additive one. Interpretability for Time Series Transformers using A Concept Bottleneck Framework is the closest methodological neighbor. It softly aligns Transformer layers, via kernel alignment, with generic concepts like an autoregressive model and hour of day, and it names the lack of domain concepts as a limitation. We differ in two ways: our bottleneck is hard, with the output a deterministic function of the concepts, and our concepts are physical. No existing work in the file applies an inherently interpretable neural model to net-load disaggregation.

Finally, the censored-meter model of Disaggregation of household solar energy generation using censored smart meter data motivates the censoring experiments. It fits Gamma-distributed demand using densities for positive readings and cumulative probabilities for zero readings. It works well at moderate capacity, but its error rises when midday readings flatline at zero for long stretches.

Research Program

We will build the model in PyTorch with a differentiable PV Watts implementation. The static quantities are parameterized so that they stay within physical ranges: capacity is positive, tilt lies between 0 and 90 degrees, and azimuth is circular. Attenuation passes through a sigmoid. Training follows the supervised setting of the Chen et al. papers and uses gross solar measured separately in the Ausgrid data. We add a weak smoothness penalty on attenuation during clear-sky periods so that it does not silently absorb demand.

Under censoring, net-load readings clamped at zero enter the loss as inequality constraints. The requirement is that predicted demand minus predicted solar be at most zero, which mirrors the density/cumulative split of the Gamma approach while keeping a learned demand structure. The static quantities are shared across time and are therefore estimated from shoulder hours that are not censored. Our hypothesis is that this lets the model carry the solar curve through clamped middays, where a free-form network has no signal.

We will also run three analyses. First, we check concept recovery: whether learned tilt and azimuth match registry metadata. We test this on French systems that have both ground-truth generation and metadata from the BDPV and BDAPPV sources, fitting the solar branch alone. Second, we test intervention value by replacing learned concepts with registry capacity, with an oracle attenuation computed as measured output divided by the clear-sky expectation, and with attenuation borrowed from sub-metered homes in the same Ausgrid postcode. Third, we check faithfulness: setting attenuation to zero should remove the solar estimate exactly, and we will demonstrate this on simulated inverter trips.

Baselines

The naive reference applies PV Watts with regional irradiance, a fixed default tilt and north-facing azimuth, and capacity set from the home's peak observed export. It shows how much any learning adds beyond textbook physics. It uses the same weather inputs as every other model, and our model must beat it at every data fraction to justify itself.

The strongest existing method is the personalized federated Transformer. The file reports $R^2 = 0.7757$ for it on Ausgrid with irradiance from the National Solar Radiation Database, and $R^2 = 0.5569$ for local-only models in the cold-start setting with 8% of the data (Privacy-Preserving Personalized Federated Learning for Distributed Photovoltaic Disaggregation Under Statistical Heterogeneity). Alongside it we run the multi-scale attention network, for which the file gives only rankings ("lowest MAE and RMSE in Summer, Winter, and Spring"). There is no reported numerical result on Lacuna for that model, so we reimplement it. Both are rerun on our splits, with identical inputs and the same hyperparameter search budget in GPU-hours. We would count parity as landing within 0.03 $R^2$ of the stronger one at full data. We would count a win as a clear margin at 8% data and under censoring.

For censored settings, the Gamma model is run on its own constructed UK dataset, which combines London smart meters with Sheffield Solar profiles, and on censored Ausgrid. The file reports capacity MAPE of 13% on the UK set and 29% on Pecan Street (Disaggregation of household solar energy generation using censored smart meter data). Beating it means lower capacity MAPE, most importantly in the high-capacity homes where it fails. For UK weather we use the atmosphere-monitoring irradiance used in Remote-Sensing-Based Estimation of Rooftop Photovoltaic Power Production Using Physical Conversion Models and Weather Data.

For interpretable competitors, we run a Transformer with soft kernel-alignment concept supervision following Interpretability for Time Series Transformers using A Concept Bottleneck Framework, with irradiance and clear-sky profiles substituted as concepts. We also run a Neural Additive Model over the same inputs.

Three ablations isolate the key ingredient. The first freezes attenuation as a per-home constant fitted by least squares, which amounts to a static physics fit. The second adds an unconstrained residual channel from the encoder to the output, recreating the leaky bottleneck. The third replaces PV Watts with a learned mapping from the same concepts.

Why not just…?

Why not just fit PV Watts per home by least squares and call it interpretable? This is the most serious threat, and it is exactly our first ablation. A static fit captures capacity and orientation but treats every departure from regional irradiance as demand. Local clouds therefore leak into the demand estimate. If the static fit matches our model on Ausgrid RMSE, then the learned attenuation adds nothing, and we would report the simpler model as the recommendation. We expect the gap to be concentrated in partly cloudy intervals, so we will report error stratified by clear-sky index.

Why not just use a black-box Transformer and explain it with SHAP or attention maps? Post-hoc attributions describe a model's behavior without constraining it. They are known to be unstable (Concept Bottleneck Models). More practically, they offer no handle for an operator who knows an inverter is offline. Our intervention experiment quantifies what that handle is worth. A black box can only be retrained or fed the information as an extra input, and we test the extra-input version too, giving the Transformer the registry capacity as a token.

Why not just use linear regression on Fourier harmonics and lags? The model in Interpretable Net Load Forecasting Using Smooth Multiperiodic Features forecasts aggregate net load and has no weather input, so it cannot separate solar from demand. We include it only as a demand-side sanity check, not as a disaggregation competitor.

Why not just use the Gamma censored-meter model? It is fast and needs no weather data, but its reported error rises to 29% MAPE precisely when capacity exceeds base load. Our censored-Ausgrid experiment, which sweeps the fraction of clamped daytime readings, will show whether shared physical structure closes that gap.

Why not just detect panels from aerial imagery? It sees only visible, mapped panels, never demand, and its metadata correlation drops to 0.61–0.68 across imagery sources (A crowdsourced dataset of aerial images with annotated solar photovoltaic arrays and installation metadata). We treat its output as one possible intervention rather than a replacement.

Proposed Experiments

The primary experiment trains every model on Ausgrid at 100%, 25% and 8% of training data. It reports per-home $R^2$, RMSE and MAE of gross solar, broken down by season. The second experiment censors Ausgrid net load at zero export and reports solar RMSE and capacity MAPE as the censored fraction grows, repeated on the constructed UK set. The third experiment trains on one group of Ausgrid postcodes and tests on unseen postcodes to measure transfer. The fourth measures intervention value as the error reduction from replacing learned concepts with oracle, registry-derived and neighbor-derived values, compared against a Transformer given the same information as input. The fifth checks concept recovery against French metadata, as circular error for azimuth and absolute error for tilt.

The decisive checks are four. The model must not trail the Transformer by more than the stated margin at full data. It must beat the static-attenuation ablation on cloudy intervals. It must beat the Transformer at 8% data and under heavy censoring. Neighbor-derived attenuation must reduce error for homes without sub-meters. If the model fails the first and second checks together, the proposal's premise is wrong. If it passes them but gains nothing from intervention, the interpretability is real but operationally idle, and we would say so.