Physics-Constrained Thermal Video Restoration for Building Virtual Batteries

Buildings can act as batteries. If you turn the heat pump down for an hour, the walls and floors keep the rooms warm for a while, and grid operators can sell that slack as "virtual energy storage." How much slack a building holds depends mostly on how much heat its walls store and how quickly they lose it. Those quantities are rarely measured, especially in old building stock. A thermal infrared camera pointed at a facade sees wall surface temperatures directly. Outdoors, though, the camera sees through haze, rain, blur and sensor noise. Some of these corruptions do more than make the picture ugly. They shift and rescale the apparent temperature, which is exactly the quantity the battery estimate depends on. Current image restoration networks are tuned to make pictures look clean, measured by PSNR (peak signal-to-noise ratio, a pixel-level fidelity score) and SSIM (structural similarity). Nothing guarantees they preserve the slow, absolute temperature trends that heat-transfer physics cares about.

We propose to restore thermal video and estimate a building's virtual battery in a single model. The model splits each frame's corruption into two parts. The first is a scene-wide radiometric part: atmospheric transmission and path radiance, shared by every wall region in the same frame. The second is a local part: rain streaks, noise and blur. A learned restorer handles only the local corruption. The scene-wide distortion and the wall temperatures are estimated jointly under a resistor-capacitor (RC) heat-transfer model, driven by logged HVAC power and outdoor temperature. Two signals can then be told apart. Thermal dynamics are region-specific and follow the HVAC inputs. Atmospheric distortion moves all regions together and ignores the HVAC. Success is measured by how accurately the system recovers battery capacity and power limits, and by how well a controller dispatches the building using them. PSNR is not the target.

Core Claim

Restore-then-measure pipelines bias virtual battery estimates in a specific way. A learned restorer "enhances contrast" by rescaling intensities from frame to frame, and the RC fit then absorbs that rescaling as fake changes in wall heat storage. A model that hands scene-wide radiometric distortion to the physics layer, rather than to the network, should recover wall capacitance and battery capacity more accurately than either of two alternatives. One is an off-the-shelf all-in-one restorer followed by system identification. The other is a filtering approach that skips restoration and simply averages raw pixels. We also claim that, under realistic haze, envelope-aware battery estimates from images beat estimates built only from indoor air sensors and HVAC logs. If that second claim fails, the camera adds nothing, and we will report that outcome.

This is a downstream version of the "More is Less" result from PRISM. That paper restores scientific images with a prompted diffusion model and finds that full blind restoration cut microscopy segmentation mIoU (mean intersection-over-union) to 0.475, against 0.580 for selective restoration. Here the downstream task is a thermodynamic estimate rather than segmentation. The restriction on the restorer is also built into the model structure, not chosen through a prompt.

Research Program

The physical side builds on existing virtual battery work. Luo et al. model each house with two heat-storage states, indoor air and structural walls, and show that ignoring wall storage misstates curtailment capacity. This motivates our need to observe walls. Zhu et al. collapse multi-zone HVAC dynamics into one building-level "state of comfort" battery and name unknown wall RC parameters as the main obstacle to deployment. This defines the gap. PhyLSTM adds ordinary-differential-equation residuals to a recurrent network's loss to identify cooling-tower flexibility, but assumes clean sensors with synthetic Gaussian noise. We borrow its residual-loss technique.

Let $s_t$ be the vector of facade-region surface temperatures at frame $t$, with $h_t$ the hidden interior states, and $u_t$ the HVAC power and outdoor temperature. The dynamics are

$$ \begin{bmatrix} s_{t+1} \\ h_{t+1}\end{bmatrix} = A(\theta)\begin{bmatrix} s_t \\ h_t\end{bmatrix} + B(\theta)\,u_t + w_t, $$

where $\theta$ collects the RC parameters and $w_t$ is process noise. The observed thermal frame is

$$ y_t = k_t * \big(a_t\, R(s_t) + b_t\big) + r_t + n_t . $$

Here $R$ renders region temperatures to a radiance image, $a_t, b_t$ are the scene-wide transmission and path radiance, $k_t$ is a blur kernel, $r_t$ is structured occlusion such as rain, and $n_t$ is sensor noise. Two quantities leave the model. The first is $\theta$, from which battery capacity in kWh and power limits follow through Zhu's aggregation. The second is a running state estimate for the controller.

A feedforward restorer handles only $k_t$, $r_t$ and $n_t$, and is barred from rescaling global intensity. We use a blind router in the spirit of AirNet, which learns a degradation code by contrastive patch learning without needing a task label. This avoids the task-ID requirement of ECMRNet, a continual thermal-infrared restorer that grows and prunes channel groups per degradation task. The barrier on global rescaling is enforced by normalizing the restorer's output to match the input's per-frame mean and spread. The parameters $a_t, b_t$ and $\theta$ are fit by alternating minimization of two terms: an image reconstruction loss and a PhyLSTM-style dynamics residual. Diffusion restorers such as WeatherDiff need about 20 seconds for a 640×432 image, so we exclude them from the online loop. Wall dynamics evolve over minutes, so latency only matters for the online state estimate.

The program has three parts. The first is a simulator. We take CityLearn buildings, attach Luo-style two-mass envelopes, render facades with real thermal textures from M3FD and HM-TIR, and apply compound degradations. These include HM-TIR-style low contrast, blur and noise, plus time-varying atmospheric transmission and rain. The second is the joint estimator. The third is a small field deployment: one building, a roof-mounted thermal camera, and thermocouples embedded in the wall as ground truth. The field data exists because no dataset in our search pairs thermal video with building energy logs.

Baselines

The naive reference fits the RC model and virtual battery only from indoor air temperature and HVAC logs, as in Zhu et al.. That paper reports costs within 3.4% to 5.7% of the optimum given by the full RC model when parameters are known. We run it on the same buildings and weather. Beating it means lower capacity error, and that is the test of whether the camera is worth having.

The second baseline is filter-only: region-averaged raw pixels go into a Kalman smoother over the same RC model, with the scene-wide affine distortion treated as noise. It has the same inputs, the same identification code and no learned components. It is the cheapest way to use the camera at all.

The strongest restoration baseline is restore-then-fit, run with three restorers. ECMRNet is the purpose-built thermal restorer. It reports a 1.61 dB average PSNR gain over the next competitor on HM-TIR, but no absolute value appears in our file, and we give it oracle task IDs, which favors it. AirNet, the blind restorer, reports 34.90 dB on Rain100L. M2Restore, a mixture-of-experts restorer combining CNN and Mamba branches, reports 31.65 dB PSNR averaged over the All-weather benchmark, at 0.17 s per 224×224 image. All three are fine-tuned on the same synthetic thermal training split with the same GPU-hour budget.

Two ablations isolate the key ingredient. One removes the dynamics residual. The other lets the restorer adjust global intensity, which recovers ordinary restoration inside our pipeline. On the control side, the fitted batteries feed the distributed quadratic-program controller of Chen et al., which reports a 16.8% average cost reduction over rule-based control. The reference point is the CityLearn score of ZO-iRL, which reports an overall score of 0.944 against a rule-based score of 1.0. The proposal wins if it reduces capacity error relative to the best restore-then-fit variant at matched haze levels and closes most of the control-cost gap to the controller run with true parameters.

Why not just…?

Why not just use contact sensors? Indoor air sensors cannot see wall heat storage. That is the core of Luo et al.'s two-mass result and Zhu et al.'s stated obstacle. Embedding sensors in every wall of an existing building is invasive, and we do it only to create ground truth. The naive sensor-only baseline tests this directly. If image-derived estimates do not beat it under realistic haze, the proposal fails on its central claim.

Why not just average pixels and run a Kalman filter? This is the most serious threat. Averaging over a facade region removes most noise and blur, so restoration may be unnecessary for those corruptions. Averaging does not remove a scene-wide gain and offset, however, and a time-varying gain mimics a change in stored heat. The decisive experiment sweeps the strength of transmission variation. Without haze we expect the filter-only baseline to match our method. If it still matches under strong time-varying haze, the learned component is unjustified, and we will say so.

Why not just run ECMRNet or AirNet and then fit? These networks are trained for pixel fidelity, and their contrast enhancement reshapes intensity from frame to frame. PRISM shows that full restoration can harm a downstream measurement even while it improves image quality. The ablation that allows global rescaling tests whether this mechanism, rather than restorer quality, explains any gap. To keep the comparison fair, the baselines receive identical fine-tuning data and compute.

Proposed Experiments

In simulation we measure three things across haze, rain, noise and blur levels: the relative error of the capacity, power-limit and wall-capacitance estimates, and the closed-loop CityLearn ramping, peak and cost scores. We report PSNR and SSIM alongside these, to test whether pixel quality and estimate accuracy come apart, as we predict. In the field deployment we measure wall-temperature root-mean-square error against the embedded thermocouples and the error of capacity estimated from a staged HVAC setback.

The decisive checks are two crossover curves. The first plots, against haze variability, the point at which our method separates from filter-only. The second plots, against envelope heaviness, the point at which image-derived estimates beat sensor-only estimates. If neither crossover appears within realistic ranges, the cameras are not worth deploying for virtual energy storage, and that negative result is worth publishing.