Disocclusions Are Mostly Observed: Dead-Time-Aware Change-Point Flux Estimation for Single-Photon Video

Single-photon cameras record the arrival time of individual photons, often more than 100,000 times per second. After each detection the pixel goes blind for a fixed recovery period, called dead time. Existing methods that turn these timestamps into brightness assume brightness stays constant while photons are pooled. A moving object breaks that assumption at its edges. When an occluder slides away, a background pixel switches abruptly from "seeing the occluder" to "seeing the background" partway through the pooling window. Frame-based pipelines then blur the two surfaces together, or discard the pixel and send it to a generative inpainter as a "hole."

Our claim is that at single-photon time resolution, most of what a frame-based pipeline calls a disocclusion hole was actually observed for part of the frame. A pixel-level estimator can recover that background brightness from real photons if it does two things. It must model the instant of the reveal as a change point inside the timestamp stream. It must also correctly account for dead time, which hides the first background photons when the occluder was bright. Only the residue that was truly never seen should be handed to a generative model. If this holds, the job of hallucinating disoccluded content shrinks substantially. The reveal time also tells us exactly where a motion-based consistency constraint must be released.

Core Claim and Mechanism

Dead-time flux estimation is well developed for static scenes. High Flux Passive Imaging With Single-Photon Sensors introduced a closed-form estimator built on the "average time of darkness" between detections, reaching roughly $10^6:1$ dynamic range. Passive Inter-Photon Imaging built a maximum-likelihood estimator over inter-photon intervals, exceeding $10^7:1$. Both assume a stationary Poisson arrival rate over the window. That assumption is the one we relax.

For a pixel whose last detection was at $t_0$, a constant-flux model with dead time $\tau$ gives the next detection time $t$ the density $\lambda e^{-\lambda(t - t_0 - \tau)}$ for $t > t_0 + \tau$, where $\lambda$ is the incident photon rate. We replace this with a two-rate model that switches at an unknown reveal time $c$, giving the interval likelihood

$$ p(t \mid t_0, c) = \exp\!\Big(-\!\int_{t_0+\tau}^{t} \lambda(s)\,ds\Big)\,\lambda(t), \qquad \lambda(s) = \lambda_{\text{occ}} \text{ for } s < c,\ \lambda_{\text{bg}} \text{ for } s \ge c. $$

Here $\lambda_{\text{occ}}$ and $\lambda_{\text{bg}}$ are the occluder and background rates. The detector-off interval $[t_0, t_0+\tau]$ contributes nothing.

Two consequences follow directly. First, if the occluder is bright, the pixel spends a fraction $\lambda_{\text{occ}}\tau/(1+\lambda_{\text{occ}}\tau)$ of its time dead (the standard non-paralyzable dead-time count-rate relation). A reveal that lands inside a dead interval is invisible until recovery. A stationary estimator applied to the post-reveal segment, or a count-based estimator that ignores dead time, therefore biases the background flux in a predictable direction. Second, the precision of $c$ and of $\lambda_{\text{bg}}$ depends on how many post-reveal photons arrive. This sets a principled per-pixel threshold below which the pixel is genuinely unobserved and must be inpainted.

The geometric-constraint literature supplies the second half of the method, and in a specific form. TokenFlow propagates features along correspondences from the source motion field. It shows that such propagation fails once topology changes, because the motion field itself becomes invalid. WAVE describes the resulting trade-off between rigid preservation and flicker. We use a dense warp to pool timestamps across time for noise reduction, but only on the segment of each pixel's stream before or after its detected change point. The warp acts as a constraint that is released exactly where the photons say the surface changed, rather than everywhere or nowhere.

Relation to Prior Work

Three papers motivate the gap. PhotonSplat fits 3D Gaussians to raw binary SPAD frames with a Bernoulli loss. Transient NeRF fits radiance fields to lidar histograms with space carving. Single-Photon Camera Guided Extreme Dynamic Range Imaging fuses dead-time-linearized flux with a fixed-viewpoint CMOS image. None of the three models how dead time interacts with visibility changes in a moving scene. PhotonSplat and Transient NeRF also leave disoccluded voids to the 3D representation alone.

On the disocclusion side, two papers supply the downstream stage we reuse. FreeFix detects low-confidence regions with a Fisher-information mask and fills them with a frozen image diffusion model. Beyond the Visible composites user-specified reveals into diffusion latents. Both treat the hole mask as given by RGB geometry and assume low-dynamic-range color. We change what enters that stage: a smaller mask derived from photon evidence, and physically calibrated flux at its border.

GeoDrag names disocclusion as its main failure mode. That supports our premise that deformation alone cannot fill revealed regions. We are not proposing an editor, however, and do not compete with it.

One result constrains the design. Time-Resolved MNIST reports that the inter-photon timing estimator becomes unstable at 2–3 photons per pixel (about 35% versus 63% accuracy for count-based estimation at 5 mlux). Our method therefore must not apply a timing likelihood to tiny post-reveal segments. It falls back to a count-based or prior-driven estimate when the post-reveal photon count is below the identifiability threshold above. We test this explicitly rather than assume it away.

Research Program

We will build the estimator in three layers. The first layer is per-pixel change-point inference: the likelihood above, maximized over $c$ by a scan over detection times, with a penalized test for "no change." The second layer is warp-pooled estimation. Coarse flux frames give a dense motion estimate, initially by optical flow and alternatively by rendering motion from a PhotonSplat model. Timestamps are pooled along the warp separately for pre-change and post-change segments, with the change point acting as a gate on the warp constraint. The third layer is residual repair. Pixels whose post-reveal evidence falls below the threshold form the hole mask passed to a FreeFix-style frozen diffusion inpainter. That inpainter receives as context the tone-mapped, dead-time-corrected flux of observed neighbors rather than clipped values.

Data come from the passive SPAD simulator of Accurate Simulation Pipeline for Passive Single-Photon Imaging. It models photon detection probability, dark counts, afterpulsing, and dead time, and it matches SPAD23 and SPAD512² hardware counts within 1%. We will drive it with rendered moving scenes that provide ground-truth flux and ground-truth reveal times. We will sweep three factors: occluder brightness relative to $1/\tau$, background dynamic range, and occluder speed in pixels per frame. Real-data evaluation uses PhotonScenes from PhotonSplat (SPAD512² at 130,000 fps), where camera motion creates parallax disocclusions. There we evaluate by held-out-view consistency rather than ground-truth flux.

Baselines

The naive reference is count-based binning. Binary detections are summed in fixed frames with no dead-time correction, then warped with optical flow, and pixels with undefined flow are inpainted. It is what one gets by treating the SPAD as an ordinary fast camera. It should fail visibly against bright backgrounds, where counts saturate near $1/\tau$.

The strongest existing competitor is a frame-based physically correct pipeline. Timestamps are binned into short frames, and each frame is converted to flux with the inter-photon maximum-likelihood estimator of Passive Inter-Photon Imaging, or with the free-running estimator of High Flux Passive Imaging where that is more stable. Frames are then motion-compensated, and RGB-style disocclusion masks are repaired with FreeFix's frozen-diffusion procedure. This baseline is the state of each subfield chained together, so beating it isolates the value of joint reasoning. We run it on identical timestamp streams, use the same diffusion model and inpainting budget, and tune its frame length over the same grid and validation scenes used to tune our penalty. No paper reports a number on this moving-scene task; for disocclusion-region flux error on single-photon video, no reported result was found on Lacuna. For FreeFix's own extrapolation benchmarks, the file gives no exact figures. For downstream novel-view quality on static SPAD reconstruction, the reference number is PhotonSplat's 15.61 dB PSNR on its simulated SPAD-to-3D benchmark, against 14.50 dB for Deblur-GS. We will report our pipeline feeding PhotonSplat against that.

A learned competitor is a network trained on simulator output to map raw timestamp streams to flux video. It gets the same simulated training budget as our tuning set plus ten times more, so that capacity is not the reason it loses.

Two ablations isolate the key ingredients. The first keeps the change-point model but replaces the dead-time likelihood with a plain Poisson one. The second keeps dead-time estimation but drops the change point, applying the warp gate from flow occlusion masks instead. An oracle with ground-truth reveal times bounds what better change-point detection could buy.

Beating the strongest baseline means two things. Log-flux RMSE must be lower inside the ground-truth disocclusion band, with no loss outside it. The area handed to the inpainter must shrink, with lower error in that area after repair.

Why not just…?

Why not just use shorter frames everywhere? Shorter frames do localize the reveal better. They also reduce the photons per pixel per frame across the whole image, which is exactly the regime where Time-Resolved MNIST shows timing estimators destabilizing. The change-point model adapts its window per pixel and only shortens it where the surface actually changed. This is a real threat, so the frame-length sweep is part of the strongest baseline. If some fixed frame length matches us across the brightness and speed sweep, the change-point machinery is unnecessary, and we will say so.

Why not just ignore dead time and count? Counting saturates at a rate near $1/\tau$, as the photocount-statistics literature summarized by On the sum-rate capacity of poisson multiple access channel with non-perfect photon-counting receiver describes. Our analysis further predicts that dead time inherited from a bright occluder hides the earliest background photons. The Poisson-likelihood ablation tests this directly: it should show a signed bias at reveals that grows with $\lambda_{\text{occ}}\tau$. If no such bias appears, the dead-time component is not pulling its weight.

Why not just let a generative model fill the holes, or train an end-to-end network? Diffusion inpainting has no notion of absolute irradiance, and Beyond the Visible documents ghosting when stochastic fills evolve across frames. Large video generators score 0.02 against 0.72 on that paper's motion-accuracy measure. A simulator-trained timestamp network is the more serious alternative. It could implicitly learn change points and dead time. We include it as a baseline and also test it on out-of-distribution dead times and brightness ratios, where an explicit likelihood should degrade more gracefully. If the network wins even out of distribution, the contribution reduces to the simulator benchmark and the analysis of identifiability.

Decisive Experiments

The first test is a controlled simulation sweep. We vary $\lambda_{\text{occ}}\tau$ from 0.01 to 10, background-to-occluder flux ratio from $10^{-3}$ to $10^{3}$, and occluder speed from 0.1 to 5 pixels per frame. On each grid cell we measure log-flux RMSE in the reveal band, fraction of the frame-based hole mask recovered from real photons, and reveal-time error. The central prediction is a specific pattern. The gap over the frame-based baseline should grow with occluder brightness, because dead-time masking worsens. It should also grow with occluder speed, because more of each frame is post-reveal. It should vanish for slow, dim scenes.

The second test checks the identifiability threshold. Recovered-pixel error should rise sharply below the predicted post-reveal photon count. If error stays low below the threshold, the threshold is too conservative; if it rises above the threshold, the model is wrong.

The third test is downstream. We feed corrected flux video and reduced masks into PhotonSplat on PhotonScenes and on simulated scenes. We then compare held-out-view PSNR and LPIPS against PhotonSplat trained on frame-based flux, with FreeFix repair applied identically to both. The claim fails if hole area shrinks but downstream quality does not improve. That outcome would mean the recovered pixels are too noisy to help a 3D model.