Lamp Flicker as an Albedo-Free Signal for Removing Baked-In Lighting

When a 3D reconstruction system learns an object's appearance from photographs, it tends to paint shadows and highlights directly onto the object's color. This is called baked-in lighting, and it shows up as soon as the object is moved into a new scene: the shadow stays where it was, and the shine does not follow the new light. The root cause is that a single pixel's brightness cannot say whether a surface is dark or simply poorly lit. We propose to break that ambiguity with a signal that is already present in most indoor scenes and that ordinary cameras discard: the flicker of electric lamps. Mains-powered lamps oscillate at 100 or 120 Hz. An event camera is a sensor whose pixels each fire independently whenever their own log-brightness changes, with no shared frame clock and microsecond timing. Such a camera records this oscillation at every pixel. In the logarithm of brightness, the surface color is a constant offset, so it drops out of the oscillation entirely. What remains is the fraction of each point's light that comes from each flickering lamp.

The core claim is that this per-pixel flicker depth is a direct, color-independent measurement of shading. A point in a lamp's cast shadow shows zero flicker from that lamp. A specular highlight of a lamp shows flicker from that lamp alone. Fitting a physically based scene model to these measurements by weighted least squares, while also fitting the sensor's own per-pixel quirks, should stop the reconstruction from absorbing shadows and highlights into albedo. It should do so under steady, everyday lighting and a moving camera, which is exactly the regime where existing event-based lighting estimation does not work.

Core Claim

Suppose a scene contains a steady light component and $K$ lamps whose output follows known or estimable periodic waveforms $s_k(t)$, each normalized to mean one. For a surface point $x$ with diffuse albedo $a(x)$, steady irradiance $D(x)$, and lamp irradiance $E_k(x)$ (which already includes visibility and the cosine term), the observed radiance is $I(x,t) = a(x)\,[D(x) + \sum_k s_k(t) E_k(x)]$. Taking logs and linearizing for modest flicker gives

$$\log I(x,t) \approx \log a(x) + \log\Big(D(x) + \sum_k E_k(x)\Big) + \sum_k m_k(x)\,\big(s_k(t)-1\big), \qquad m_k(x) = \frac{E_k(x)}{D(x) + \sum_j E_j(x)}.$$

The albedo enters only as a time-constant offset. The flicker depths $m_k(x)$ depend only on lighting, geometry, and visibility. A view-dependent specular term adds a lamp-$k$ reflection with the same waveform $s_k$, which pushes $m_k$ toward one inside that lamp's highlight. As the camera moves, the change in $m_k$ therefore traces the specular lobe.

An event camera fires whenever log-brightness moves by a per-pixel contrast threshold $C(u)$. Counting events per flicker cycle and demodulating at each lamp's frequency gives an estimate of $m_k(u)/C(u)$ at pixel $u$. The ratio $m_k/m_j$ between two lamps at the same pixel does not depend on the threshold either. This is the measurement we fit. It is not something a model can "explain away" by repainting albedo, because albedo does not appear in it.

Relation to Prior Work

Two event-camera papers motivate the proposal and also mark its boundary. Indoor Lighting Estimation Using an Event Camera uses the microsecond timing of the sensor to watch a lamp's power-on warm-up on a known diffuse sphere in a dark room. This breaks the intensity-distance ambiguity: it reduces distance error from 75.63 cm for an image-based method to 27.05 cm, and direction error from about 38° to 13.4°. It requires a transient, a calibration target, a single point light, and a known Lambertian surface. We keep its insight that temporal structure resolves what a static image cannot, but we use the steady flicker of lamps that are already on, on unknown objects, and we estimate materials jointly. EvGGS builds 3D Gaussian splats (scenes represented as many small colored 3D blobs) from event streams and reaches 27.95 PSNR on Ev3DS. To do so it gives each Gaussian a single grayscale intensity, which by construction bakes all shading in. We treat it as the event-only reference that our flicker term is meant to correct.

The inverse-rendering line supplies the scene model and the optimizer we build on. PhySG represents light and specular reflectance with spherical Gaussians so that rendering integrals have closed forms. It ignores indirect light and self-shadowing. TensoIR uses factorized tensors fast enough to ray-march visibility and indirect bounces during training, but it fixes dielectric Fresnel at 0.04. Extracting Triangular 3D Models, Materials, and Lighting From Images recovers meshes and texture maps with a split-sum shading approximation that omits shadowing. NeRFactor distills geometry and uses a measured-material prior under a coarse 16×32 light map. ComGS caches occlusion and indirect light in surface probes for fast object insertion and reports +1.4 dB PSNR on SynCom. Every one of these fits RGB images by minimizing a photometric squared error. That loss is exactly the one that rewards baking. We do not replace these models. We add a second residual family that the baking shortcut cannot satisfy, and we use TensoIR's ray-marched visibility as the forward model for $E_k(x)$, since cast shadows are where $m_k$ is most informative.

The crystallographic least-squares literature supplies the estimation discipline, and this is where the second direction does real work. Small-molecule refinement minimizes $\sum w\,(F_{\text{obs}}^2 - F_{\text{calc}}^2)^2$, with weights $w = 1/[\sigma^2 + (aP)^2 + bP]$ that combine counting statistics with empirical terms, as in the refinements reported in the dimanganese azido complex and the iridium complex. Serial femtosecond crystallography post-refinement, reviewed in XFEL data analysis for structural biology, jointly refines per-snapshot nuisance parameters (orientation, partiality, scale) alongside the structure. That review reports the number of required patterns falling from hundreds of thousands to a few thousand. Our event-derived flicker depths have the same character. They are counts with Poisson-like noise, scaled by an unknown per-pixel threshold that varies across the sensor, a known event-camera weakness noted in both event papers. We therefore refine $C(u)$ as a nuisance scale and adopt the same weighting form, with $P$ built from predicted event counts. The known non-convexity of crystallographic refinement also constrains the design: Gauss-Newton started far from the answer falls into false minima. So we initialize lamp positions from a closed-form step, by triangulating shadow boundaries where $m_k$ drops to zero, before refining anything.

The triggerless-readout literature supplies the systems pattern. In LHCb's Run 3 trigger (A Comparison of CPU and GPU Implementations for the LHCb Run 3 Trigger), reconstruction placed directly on the streaming GPUs cut downstream bandwidth by a factor of 30 to 60. Its core is recursive least squares (Kalman filtering) on unbuffered hits. Flicker demodulation is purely local to each pixel. It can run on the raw event stream as a per-pixel phasor accumulator, reducing millions of events per second to a few complex numbers per pixel per time window. The refinement then consumes only these sufficient statistics.

Research Program

We will build four things. The first is a demodulator that turns raw events into per-pixel, per-window flicker phasors at each lamp frequency, with variance estimates. Lamp frequencies come either from known mains frequency and harmonics or from a programmable driver in controlled experiments. The second is a forward model that, given geometry, lamp positions, a steady environment term, and spatially varying albedo, roughness, and metallicness, predicts $m_k(x,\omega_o)$, including the specular contribution. This uses TensoIR's ray-marched visibility, extended with a refinable Fresnel term so that metals are not forced to the 0.04 dielectric value. The third is a joint objective: the usual photometric loss on intensity, taken from a hybrid sensor's low-rate frames or from event-reconstructed intensity, plus a weighted flicker term

$$\mathcal{S}_{\text{flk}} = \sum_{u,\tau,k} w_{u\tau k}\,\Big(\hat{n}_{u\tau k} - \tfrac{1}{C(u)}\, m_k\big(x(u,\tau),\omega_o(\tau)\big)\Big)^2,$$

where $\hat{n}_{u\tau k}$ is the demodulated event amplitude for lamp $k$ at pixel $u$ in window $\tau$, $C(u)$ is the refined threshold, and $w$ follows the crystallographic weighting form. The fourth is a staged solver: shadow-boundary initialization of the lamps, then Gauss-Newton or Adam refinement, then an R-factor-style residual index computed separately on held-out views to flag remaining baking.

The analysis question we care most about is when the flicker term actually separates albedo from shading. We expect three regimes. With one flickering lamp and no steady light, $m_k \equiv 1$ everywhere and the term carries no information. With a lamp plus daylight, $m_k$ maps that lamp's share of irradiance. With multiple lamps at distinct phases or frequencies, ratios between lamps remove the threshold as well. We will report the regimes explicitly rather than claiming the method works everywhere.

Baselines

The naive reference point is the same scene model and optimizer trained with the photometric loss alone, which amounts to TensoIR run on intensity images from the same capture. It receives identical data and iteration budget, and the flicker term is simply switched off. If adding $\mathcal{S}_{\text{flk}}$ does not improve albedo and relighting error over this, the proposal fails at its central claim.

The strongest existing frame-based method is TensoIR, run in its published configuration on the frames, with its own hyperparameter search given the same GPU-hour budget as ours. The file reports that TensoIR outperforms NeRFactor and InvRender in normal error and albedo PSNR, but I found no reported numerical value on Lacuna for its albedo PSNR, so we will rerun it ourselves rather than quote a number. For the composition task we compare against ComGS. It reports a +1.4 dB PSNR improvement over GS-IR and IRGS on SynCom, and it is the method most directly aimed at baked shadows in object insertion. Beating it means a higher composition PSNR on flicker-rendered SynCom scenes at matched reconstruction time. On geometry, PhySG's reported normal error of 2.528° on its glossy synthetic set is the reference figure for normals.

The event-only reference is EvGGS at its reported 27.95 PSNR on Ev3DS. We will retrain it on our flicker-rendered scenes. It should match us on novel-view PSNR under the capture lighting and fail on relighting, and that contrast is the point of the comparison.

Three ablations isolate the key ingredient. The first replaces the albedo-free flicker depth with absolute event-reconstructed intensity at the same temporal resolution, which tests whether the albedo cancellation matters rather than just extra data. The second fixes $C(u)$ to a global constant, which tests whether nuisance refinement matters. The third uses uniform weights instead of the counting-statistics weights.

Why not just…?

Why not just capture bracketed HDR frames from many views? HDR fixes saturated highlights but does nothing about the core ambiguity: a static image cannot tell a dark surface from a dim light. Chen et al.'s comparison makes the cost concrete, with the image-based GH19 at 75.63 cm distance error versus 27.05 cm from temporal event cues (Indoor Lighting Estimation Using an Event Camera). Our photometric-only baseline is, in effect, this alternative, run with HDR frames. The albedo and relighting gap between it and the full method settles the question.

Why not just use a high-speed frame camera to see the flicker? This is a real threat. A camera running at several kilohertz can also demodulate 100 Hz flicker, and its measurements carry color. Its disadvantages are the shared exposure window, limited dynamic range at short exposures, and the bandwidth of full frames, the same pressures that pushed physics detectors toward triggerless readout. We will test it directly by simulating a high-speed rolling-shutter camera with matched photon budget and bit depth on the same scenes, feeding its demodulated flicker into the identical objective. If it matches the event camera, the honest conclusion is that flicker, not the event sensor, is the contribution, and we will report it that way.

Why not just switch lights on one at a time with an ordinary camera? Time-multiplexed lighting is the classical way to separate lamp contributions, and in a static lab it may beat us. It requires controlling the lights and a static scene, and it breaks the steady appearance people actually live in. Our programmable-flicker lab setup will include a one-light-at-a-time capture as an upper bound. The claim we defend is narrower: approaching that bound with all lights on, with a moving camera, and in passive mains-flicker scenes where switching is not possible.

Proposed Experiments

Synthetic experiments come first, because ground-truth albedo and relighting exist only there. We will re-render the NeRF-synthetic objects used by TensoIR and the 16 SynCom compositions with time-modulated emitters plus a steady environment, then convert high-frame-rate renders to events with per-pixel threshold noise. Metrics are albedo PSNR, normal mean angular error, relit-image PSNR under held-out lighting, composition PSNR on SynCom, and a shadow-leakage score: the correlation between recovered albedo and ground-truth lamp visibility, which should be near zero if shadows are not baked in. We will sweep the number of lamps, the steady-light fraction, the flicker modulation depth, and the threshold noise to map the regimes described above.

Real experiments use a hybrid event-plus-frame camera on a handheld path around objects under two setups. In the controlled setup, LEDs are driven at distinct inaudible, invisible frequencies; ground truth comes from one-light-at-a-time capture and from re-photographing objects under new lamp positions. In the passive setup, ordinary mains-lit rooms are used, in the spirit of Chen et al.'s wild indoor collection, for which the file reports no numeric result. There we evaluate by predicting the photograph after a lamp is physically moved. We will also measure throughput of the on-stream demodulator and the data reduction from raw events to phasors.

The decisive check is the first ablation on the real controlled data. If the albedo-free flicker depth beats absolute event intensity at equal temporal resolution on relighting error and shadow leakage, the albedo cancellation is doing the work. If the two tie, the gain came from more data rather than from the signal's structure, and the idea should be abandoned in its current form.