When a 3D model of an object is built from a few infrared photographs, two things change between shots that a 3D generator never expects. The object itself warms or cools during capture, and the camera's own sensor drifts as its electronics heat up. Generators that fill in unseen views assume the scene stays fixed. They treat these differences as disagreements about shape or surface appearance, and they resolve them the way they resolve any multi-view conflict: by averaging into semi-transparent clutter, or by inventing duplicated features. In thermal imagery the duplicated feature is a hot spot. A warm face appears on the back of a head, or a second engine glows at the rear of a vehicle. This is the thermal version of the "Janus problem", the multi-faced artifact of 2D-to-3D generation.
We propose a lifting stage for sparse-view thermal 3D that splits every thermal observation into three parts. The first is a static geometry. The second is a per-surface temperature that follows Newton's law of cooling over capture time. The third is a low-dimensional, slowly varying sensor drift term for each view. Any view proposed by a generative model must be explained by this factored model, or it is down-weighted. The claim is that most thermal Janus artifacts in sparse capture come from misattributed drift, not from a missing shape prior. A physically factored consistency check should therefore remove them more cheaply and more reliably than generic multi-view consistency methods built for RGB.
Core Claim and Positioning
Work at this intersection supplies pieces but not the combination. Three thermal reconstruction systems supply the physics and the data, but none generates unseen content. ThermalGaussian fits 3D Gaussian splats (a scene made of many small, colored, semi-transparent ellipsoids) jointly to RGB and thermal images. It assumes a static thermal scene and dense views. ETGS attaches a first-order cooling equation to each Gaussian and reaches 40.68 dB average PSNR on its Rapid Heat Dynamics (RHD) dataset, against 34.96 dB for the data-driven NTR-Gaussian. It requires dense, synchronized views and has no prior for surfaces it never saw. We reuse its per-Gaussian cooling model as a component. TDiff removes fixed-pattern noise and sensor drift from uncooled microbolometers, but it works one 2D frame at a time.
Three RGB generation methods enforce consistency, but none of them separates time or sensor effects from geometry. MVD-Fusion reprojects features using predicted depth while it generates. CAT3D generates many pose-conditioned views at once. Carve3D fine-tunes a generator with a reward that reconstructs and re-renders the generated views.
Two more papers motivate the failure mode. Kim et al. and Voynov et al. document capture rigs whose depth biases and calibrations drift with temperature until roughly an hour of warm-up has passed.
What is new here is the consistency test. Generated views are judged against a model in which temperature may legitimately change with capture time, and the sensor may legitimately shift by a smooth per-view gain and offset. Disagreement that neither effect can explain is attributed to hallucination.
Research Program
We take an important simplification directly from the strongest skeptical objection. Geometry comes from the paired RGB stream, using an off-the-shelf consistent RGB pipeline such as CAT3D or MVD-Fusion. Shape-level Janus is already well handled in RGB, and thermal images are too textureless for reliable pose or shape estimation, as ThermalGaussian reports. The open problem is the thermal channel on surfaces the infrared camera saw only sparsely, at different times, through a drifting sensor.
On this fixed geometry, each Gaussian $i$ carries a temperature $T_i(t)$ that obeys ETGS's cooling equation. The equation includes a heat-source term $Q_i$, and we place a sparsity penalty on it, so that new heat sources must be paid for.
Each thermal view $k$, captured at time $t_k$, is modeled as follows:
$$I_k(p) = a_k\, \mathcal{R}_k[T(t_k)](p) + b_k + F(p) + \epsilon,$$
The symbols are defined as follows:
- $\mathcal{R}_k$ renders per-Gaussian temperatures into view $k$.
- $a_k$ and $b_k$ are the view's sensor gain and offset. They are tied across consecutive views by a smoothness penalty in time, which reflects warm-up curves that are slow but not static.
- $F$ is a fixed pattern shared across all views from the same sensor.
- $\epsilon$ is residual error.
A generative proposer then produces thermal views for unobserved poses. This proposer is an RGB-conditioned, pose-conditioned image diffusion model, fine-tuned on paired RGB-thermal frames. Each proposed view is scored by the residual it leaves after a few joint fitting steps of the factored model. Proposals are reweighted by that residual, which works like rejection sampling, before final fitting. This deliberately avoids Carve3D-style RL fine-tuning. The scoring needs only a few fitting steps on an already reconstructed scene, not a fresh reconstruction for every policy update.
Baselines
The naive reference fits ThermalGaussian directly to the sparse thermal views on the RGB-derived geometry, with no generator and no drift model. It shows how much of each unseen surface is simply undetermined.
The strongest physics competitor is ETGS run on the same sparse subset, with identical geometry initialization and iteration budget. On dense RHD it reports 40.68 dB average PSNR. For sparse-view thermal reconstruction there is no reported result found on Lacuna for ETGS or any other method, so we report both settings ourselves.
The strongest generative competitor is the same thermal proposer lifted with a static-consistency check. This check uses reconstruction-and-rerender agreement in the spirit of Carve3D's multi-view reconstruction consistency (MRC) score, which Carve3D reports reducing from 0.0892 to 0.0606 on the 415-prompt DreamFusion set. It uses the same proposals, the same number of fitting steps, and the same rejection budget. It differs from our method only in lacking time and drift terms. Beating it means lower held-out thermal error and fewer phantom hot spots at matched compute.
A further competitor applies TDiff to every frame before lifting. Since TDiff reports about 165 s per image, we compare on quality and report wall-clock time separately.
Three ablations each remove one ingredient: the per-view drift terms, the cooling equation (a static temperature field), and the heat-source sparsity penalty. These isolate which part does the work.
Why not just…?
Why not just normalize each thermal frame, by histogram matching or flat-fielding, and then use an RGB-style consistency pipeline? This is the most serious threat. It removes a global offset at almost no cost. However, it cannot tell sensor drift from real cooling of the object, so in scenes that are genuinely cooling it should erase true temperature change. We test it on the cooling and heating scenes of RHD, measuring absolute temperature error on held-out views. If normalization matches our method there, the drift terms are not needed, and we would say so.
Why not just paint thermal texture onto the RGB geometry by projecting the captured frames? Projection is part of our base, but it leaves unobserved surfaces empty. Inpainting those surfaces view by view brings back the cross-view divergence that In-N-Out attributes to independent denoising trajectories. The naive baseline and the static-consistency baseline measure exactly this gap.
Why not just run ETGS on the sparse views? ETGS models physics well but has no prior for surfaces it never saw. Running it as a baseline on identical sparse subsets will show whether its cooling model alone suppresses phantom hot spots or simply leaves those surfaces undetermined.
Why not just fine-tune the generator with a consistency reward, as Carve3D does? Carve3D's reward has no notion of time or sensor drift. It would penalize physically correct cooling as inconsistency, and it requires a full reconstruction inside every update. Our static-consistency baseline is a compute-matched stand-in for that reward.
Proposed Experiments
We start with a diagnostic. We take dense, pixel-aligned RHD scenes and the static RGBT-Scenes captures, subsample them to three to eight thermal views spread across capture time, and hold out the remaining views as ground truth. Into the static scenes we inject controlled per-view gain and offset drift that follows realistic warm-up curves. We then measure how phantom hot spots and consistency scores grow with drift magnitude under the static-consistency baseline. If artifacts do not grow with injected drift, the core claim fails early.
The main comparison reports three metrics on held-out views:
- thermal PSNR and absolute temperature error;
- an MRC-style consistency score;
- a phantom hot-spot rate, defined as the fraction of unobserved surface area whose fitted temperature exceeds the dense-capture reference by a fixed threshold.
The decisive checks are two. First, our method should beat the static-consistency baseline as injected drift increases, while matching it at zero drift. Second, on genuinely cooling RHD scenes, it should beat per-frame normalization on absolute temperature error. Failing either check would indicate that thermal Janus is a texture-prior problem rather than a drift-attribution problem.
References
- ThermalGaussian: Thermal 3D Gaussian Splatting
- ETGS: Explicit Thermodynamics Gaussian Splatting for Dynamic Thermal Reconstruction
- TDiff: Thermal Plug-And-Play Prior with Patch-Based Diffusion
- MVD-Fusion: Single-view 3D via Depth-consistent Multi-view Generation
- CAT3D: Create Anything in 3D with Multi-View Diffusion Models
- Carve3D: Improving Multi-view Reconstruction Consistency for Diffusion Models with RL Finetuning
- Design and calibration of a multi-view TOF sensor fusion system
- Multi-Sensor Large-Scale Dataset for Multi-View 3D Reconstruction
- In-N-Out: Lifting 2D Diffusion Prior for 3D Object Removal via Tuning-Free Latents Alignment