Counting Vibration Modes in the Dark: Noise-Aware Rank Selection for Physical Parameter Estimation from Low-Light Video

When a deformable object vibrates, its motion is mostly a sum of a few natural vibration patterns, called modes. Each mode has a frequency set by the material's stiffness, and only some modes get excited in any given event. Measuring stiffness and damping from video therefore requires answering two questions together: which modes are actually present, and what material parameters produce their frequencies and decay rates. We propose to answer both from dim, noisy raw video. A differentiable modal simulator renders the expected photon count at every pixel. Each candidate mode carries its own Bayesian "relevance" variable that can switch the mode off, a technique known as automatic rank determination. The fit uses the actual photon-counting noise model of the sensor rather than a squared-error loss.

The central claim is that, under low light, rank selection and the noise model cannot be separated. With a squared-error loss, the pruning mechanism keeps spurious modes in bright regions, where shot noise is large, and kills real but faint modes in dark regions. With a calibrated Poisson-Gaussian likelihood, the number of surviving modes should follow a predictable curve as the photon budget shrinks. Stiffness estimates should then degrade gracefully instead of diverging. Physics-from-pixels methods have only been shown on clean imagery, and rank-pruning methods have only been shown under Gaussian noise, so this regime is untested by both.

What exists and what is missing

Differentiable physics can recover physical constants from pixels when the images are clean. gradSim couples finite-element mechanics with a soft rasterizer and reports a mass absolute relative error of 9.01×10⁻⁵, two orders of magnitude better than a ConvLSTM baseline. Its authors flag sensor noise and illumination as unresolved. PAC-NeRF does the same with particle-based continuum simulation and radiance fields, recovering a viscosity of 209 against a true 200. It needs clean multi-view video and a hand-chosen particle budget. Bayesian Differentiable Physics for Cloth Digitalization puts a posterior over cloth stiffness, but only from static silhouettes. These papers supply our forward-model machinery and motivate the gap: none handles photon-starved input, and all fix the number of degrees of freedom in advance.

Rank determination supplies the pruning mechanism. Bayesian CP Factorization with Automatic Rank Determination gives each component a shared precision with a Gamma prior. Components that do not explain the data have their precision driven to infinity, and the method recovers the true rank in 100% of synthetic trials under Gaussian noise. Factor Group-Sparse Regularization achieves the same effect deterministically with column-norm penalties and tolerates fivefold over-ranked initialization. The generalized hyperbolic prior is a cautionary result: Gamma-based priors can wrongly extinguish weak true components when noise is strong. That is exactly the failure we expect in dark pixels, and it constrains our design.

The nearest dynamical analogues have the same blind spots. Φ-ROM fixes its latent dimension by hand and observes state directly rather than pixels. Rank-revealing latent-ODE tensor decomposition prunes temporal components but uses unconstrained neural dynamics and has never seen sensor noise.

On the imaging side, RawNeRF shows that fitting a 3D model directly to linear raw data beats fitting to processed images: 23.53 dB versus 19.43 dB sRGB PSNR for a standard NeRF, and versus 22.75 dB for the multi-image denoiser UDVD. Its scenes are static. Learning Physics-Informed Noise Models from Dark Frames supplies the calibration recipe for shot noise, read noise and dark current, which we reuse. Our proposal differs from all of these in one respect: the noise model sits inside the evidence that decides how many physical modes exist.

Core Claim

We restrict to elastic objects with a known rest mesh and known camera, observed by a single static raw camera in dim light.

Forward model. From the mesh we compute $M$ candidate vibration modes by finite-element modal analysis. The displacement of vertex $v$ at time $t$ is

$$u_v(t)=\sum_{k=1}^{M} a_k\,\phi_{k,v}\,e^{-\zeta_k\omega_k t}\cos(\omega_k^{d} t+\varphi_k),$$

where the symbols are as follows:

  • $\phi_{k,v}$ is mode $k$'s shape at vertex $v$.
  • $a_k$ and $\varphi_k$ are the mode's amplitude and phase.
  • $\omega_k$ is the mode's frequency. For a homogeneous isotropic body, $\omega_k$ scales with $\sqrt{E/\rho}$, so Young's modulus $E$ is identifiable given density $\rho$.
  • $\zeta_k$ comes from two Rayleigh damping coefficients.
  • $\omega_k^d$ is the damped frequency.

A soft rasterizer turns the displaced mesh into expected photon counts $\lambda_{i,t}$ per pixel. A scene illumination gain is co-optimized, so brightness is never "fixed" by a 2D enhancer before fitting.

Noise and pruning. Each amplitude gets a Gaussian prior $a_k\sim\mathcal{N}(0,\gamma_k^{-1})$, with a Gamma hyperprior on the precision $\gamma_k$, following the Bayesian CP construction. Observed raw values follow a Poisson-Gaussian model with read noise and black level calibrated from dark frames. At very low counts we evaluate the likelihood by a truncated sum over photon counts; above a few tens of photons we use the variance-$(\lambda+\sigma_r^2)$ Gaussian approximation. Variational inference over amplitudes and precisions runs jointly with gradient-based optimization of $E$, the damping coefficients and the gain.

Testable prediction. Consider how much information the video carries about one mode's amplitude:

$$\mathcal{I}_k=\sum_{i,t}\frac{(\partial\lambda_{i,t}/\partial a_k)^2}{\lambda_{i,t}+\sigma_r^2}.$$

Here the sum runs over pixels $i$ and frames $t$, and the denominator is the local Poisson-plus-read-noise variance. The standard single-component analysis of evidence maximization says a component survives roughly when its squared amplitude times its information exceeds a constant. This is the "signal beats noise for that mode" condition. Two consequences follow:

  • The retained set of modes should shrink as the photon budget falls, at budgets we can compute in advance from the ground-truth scene.
  • A squared-error loss uses a constant denominator. It mis-weights modes whose motion lives in bright versus dark image regions, and so it should fail this prediction in a specific, signed way.

Research Program

Benchmark. We first build a low-light modal benchmark. Deformable objects from the gradSim deformable set and elastic scenes in the style of PAC-NeRF are re-simulated with modal dynamics. In each event, a randomized impact excites a random subset of modes, not necessarily the lowest ones. The scenes are rendered linearly and degraded with noise parameters calibrated from real dark frames, at mean budgets from about a thousand photons per pixel down to about one. Ground truth includes $E$, the damping coefficients, and the exact set of excited modes.

Real capture. We then capture a small real set of cantilevered plates and beams struck under dim light with a raw-capable camera. Reference natural frequencies and decay rates come from a contact accelerometer. A well-lit repeat of each strike serves as a clean-video reference.

Analyses. We test three things:

  1. The predicted retained-rank-versus-photon-budget curve.
  2. Parameter accuracy and divergence rate.
  3. Variance of the stiffness gradient across noise realizations. This tests the hypothesis that pruning spurious modes is what stabilizes backpropagation, rather than merely the likelihood.

We also swap the Gamma hyperprior for a generalized hyperbolic prior to see whether faint true modes in dark regions are better preserved.

Baselines

Naive reference. The do-nothing reference is the gradSim recipe applied as is. Frames are converted through a standard camera pipeline to sRGB, and all $M$ modes are fit with squared error, using the same rasterizer, optimizer and iteration budget as ours. On clean video this recipe yields gradSim's reported 9.01×10⁻⁵ mass error. No reported result was found on Lacuna for any differentiable-physics parameter estimation under low-light noise, so the naive pipeline's degradation curve is itself a first measurement.

Enhance-then-simulate. The strongest competitor is a cascade. Raw video is first restored by a learned raw-to-sRGB video network trained as in Seeing Motion in the Dark, which reports 28.26 dB static PSNR on its Dark Raw Video benchmark. We also test the lightweight enhancers SCI and Zero-DCE as front ends. The restored frames are then fit with fixed-rank squared error. For fairness, the restoration network is fine-tuned on raw/clean pairs from our own simulator at every photon budget, at a compute budget matched to our total optimization cost. The fixed rank is tuned on a held-out set of scenes.

Model-selection competitor. The second strong competitor keeps our exact Poisson-Gaussian likelihood and our simulator but replaces pruning with explicit selection. It runs separate fits for nested mode counts and picks the count by BIC or by likelihood on held-out frames, with the same total compute.

Ablations isolating the key ingredient. Three ablations sit alongside:

  • ARD with a squared-error likelihood, which separates the noise-model contribution.
  • The correct likelihood with all $M$ modes and no pruning, which separates the pruning contribution.
  • Group-sparse column penalties in the style of Factor Group-Sparse Regularization in place of Bayesian precisions.

A ConvLSTM regressor trained on our simulated data is included as the black-box reference, because gradSim's clean-data comparison is the only evidence on it.

What counts as winning. We count it as beating the field if, at budgets of ten photons per pixel or fewer, our method attains lower median relative error on $E$ and damping than the cascade and the BIC search, with no higher divergence rate, and higher F1 on the recovered mode set.

Why not just…?

Why not just denoise the raw video first and then run standard system identification? This is the most serious threat, because modern raw video restorers are strong. Two pieces of evidence argue against it. First, RawNeRF's gap of 23.53 dB versus 19.43 dB shows that a joint model on raw data beats fitting to processed images. Second, restorers trade noise for spatial smoothing, and that smoothing removes exactly the small, high-frequency displacements of higher modes. We expect the cascade to be competitive at moderate budgets and to systematically under-count high modes, biasing $E$, at low budgets. Because we fine-tune the restorer on in-domain pairs, a cascade win would be a clean negative result, and we would report it as such.

Why not just use the correct noise likelihood with all modes and skip pruning? The physics already pins each mode's frequency to $E$, so extra modes cannot fit arbitrary noise. That makes this a real threat. However, each unneeded mode still has a free amplitude and phase that can absorb correlated noise and rendering error. These free parameters inflate the variance of the stiffness gradient. The no-pruning ablation together with the gradient-variance analysis settles the question directly. If no-pruning matches ARD on parameter error and variance, the contribution reduces to the likelihood, and we would state it that way.

Why not just pick the number of modes by BIC or held-out frames? With the correct likelihood, this may work well whenever excited modes are the lowest $K$. Our benchmark deliberately excites non-contiguous subsets, for example a strike on a node of the first mode. In that case nested search must over-include modes, while exhaustive subset search grows as $2^M$. ARD selects a subset in a single run. The comparison on contiguous versus non-contiguous excitations measures exactly where this advantage holds and where it vanishes.

Why not just train a network to regress the parameters from video? gradSim's clean-data result already shows a two-order-of-magnitude deficit for ConvLSTM. A regressor also gives no mode count and no posterior. We include it only to confirm that noise does not reverse this ordering.

Proposed Experiments

Decisive check 1: rank curve. The retained-mode count from our method should track the photon-budget threshold predicted from the information criterion. The squared-error ARD ablation should deviate in the signed way described above: extra modes concentrated in bright regions, lost modes concentrated in dark regions. If both versions track the prediction equally, the coupling claim is false.

Decisive check 2: parameter accuracy. On simulated scenes we report:

  • relative error of $E$ and of the damping coefficients;
  • mode-set precision and recall;
  • vertex trajectory error;
  • divergence rate;
  • stiffness-gradient variance across 20 noise seeds per scene.

All metrics are reported at each photon budget against all baselines.

Decisive check 3: real transfer. On the real plates and beams, we check whether estimated natural frequencies and decay rates match accelerometer measurements within the spread between repeated well-lit strikes.

Expected outcome. A positive result would show that photon-limited video supports physical identification up to a computable rank, and that respecting that limit is what keeps differentiable simulation stable in the dark.