Seeing the Whole Lag: Spectral Alignment for Drifting Camera–IMU Clocks

Phones, drones and cheap robots fuse a camera with an inertial sensor (an IMU, which measures rotation rate and acceleration). Their clocks are rarely synchronized, and the offset between them wanders as the operating system stalls and the hardware warms up. Current online estimators treat the time offset as one more variable to refine. They refine it by nudging each tracked image point a small distance along its local velocity. That is a local search in time, and it works only when the error is already small: published systems hold up to roughly 30–60 ms and fail beyond about 100 ms. We propose to give the offset estimator a global view of lag instead. Over a window of a few seconds, compare the camera's rotation-rate signal with the gyroscope's rotation-rate signal at every possible lag at once, using a learned Fourier-domain correlation. The output is a full probability distribution over the offset. That distribution is handed to a standard visual-inertial optimizer, both as its starting point and as a check on whether the offset can be trusted at all.

The idea borrows from the receptive-field literature, where the problem is how a network can relate distant positions without quadratic cost. Here the "distant positions" are distant time lags, and the operator that sees all of them cheaply is the Fourier transform. The new claim is narrow and testable. A learned spectral lag search that sees every candidate lag up to ±0.5 s, and that learns which frequency bands carry reliable timing information, will do three things: recover large initial offsets, track offsets that change non-stationarily, and flag windows where motion is too weak to identify the offset. Local refinement alone cannot do these things, and an unlearned correlation does them less reliably.

Why the existing estimators are local in time

Online Temporal Calibration for Monocular Visual-Inertial Systems corrects each feature as $\mathbf{z}_l(t+t_d)\approx\mathbf{z}_l(t)+t_d\mathbf{v}_l$. This is a first-order expansion around the current offset guess, so the region where the estimate converges shrinks as image acceleration grows. The same paper assumes $t_d$ is constant within the sliding window of recent frames that the optimizer solves over. Modeling Varying Camera-IMU Time Offset in Optimization-Based Visual-Inertial Odometry relaxes the constancy assumption by letting the offset follow a Gaussian random walk. It also folds rolling-shutter readout into a single center-row delay. It remains a gradient-based refinement over sparse corner tracks, and the research file records that its initialization degrades beyond about 100 ms. Both papers motivate our work. They show that visual–inertial motion mismatch encodes the offset, and that the failure is one of capture range and non-stationarity, not of the signal itself. Online Spatio-temporal Calibration of Tightly-coupled Ultrawideband-aided Inertial Localization makes the same move in a Kalman filter, adding latency to the filter's state, and inherits the same local linearization. The learned multimodal fusion benchmarked in Benchmarking Visual-Inertial Deep Multimodal Fusion for Relative Pose Regression and Odometry-aided Absolute Pose Regression sits on the other side of the gap: it assumes pre-synchronized streams. Any residual offset would therefore silently corrupt its cross-modal attention.

The method comes from the receptive-field literature. Butterworth as Attention: Anisotropic Spectral Gating for Pansharpening shows three things we reuse. A Fourier transform gives an immediate global receptive field at $O(N\log N)$ cost. Naive convolution on spectra fails because frequency space is not translation-invariant. Classical filter parameters (cutoff, order) can be predicted per input as a learnable gate. We carry that gate from 2D image spectra to the 1D cross-spectrum between two sensor streams. CCNet: Criss-Cross Attention for Semantic Segmentation and ParCNetV2: Oversized Kernel with Enhanced Attention are the alternative ways to reach global context sub-quadratically. For a 1D lag search, a single FFT is simpler than either, so they serve as design contrasts, not components. The file also records a limitation of Butterworth-style gates: their smooth roll-off cannot represent sharp transients. This constrains us. Abrupt clock steps, such as scheduler stalls, must be handled by re-running the search window by window, not by expecting the gate to model the discontinuity.

Core Claim

Let $\boldsymbol{\omega}_v(t)$ be the camera-derived angular velocity over a window, and $\boldsymbol{\omega}_g(t)$ the gyroscope angular velocity. For each axis, we compute a gated, whitened cross-correlation over all lags $\tau$:

$$c(\tau)=\mathcal{F}^{-1}\!\left[G_\theta(f)\,\frac{\Omega_v(f)\,\Omega_g^*(f)}{|\Omega_v(f)\,\Omega_g^*(f)|+\epsilon}\right](\tau).$$

Here $\Omega_v$ and $\Omega_g$ are the Fourier transforms of the two signals, $^*$ is complex conjugation, $\epsilon$ is a small constant, and $G_\theta(f)$ is a Butterworth-form gate. A small network predicts the gate's cutoffs and orders separately per axis and per window from the two magnitude spectra. The three axes' correlations are summed and passed through a softmax over $\tau\in[-0.5,0.5]$ s, giving a posterior $p(t_d)$. With $G_\theta\equiv1$ this reduces to a classical whitened cross-correlation, which is exactly the baseline we must beat. The learned gate exists to suppress bands that carry no timing information: propeller vibration, rolling-shutter wobble, IMU noise above the camera's frame-rate limit, and the near-DC content that dominates slow motion. The sharpness of $p(t_d)$ then serves as an observability signal. When motion is degenerate the posterior should be flat, and the system should decline to update the offset.

Rotation rate is used on purpose. For a monocular camera it is independent of the unknown scale and translation. This sidesteps the coupling between offset and lever-arm, which the file identifies as a source of weak observability.

Research Program

The visual rotation signal comes from a global, not sparse, estimator, because Gap 1 in the file is that sparse corner velocities collapse under blur and low texture. We compute inter-frame rotation from log-polar phase correlation of whole downsampled frames, which is itself a global receptive-field operation. We fall back to a robust rotation fit on tracked corners when available, and compare the two. The whole signal path costs a few FFTs per window, so it runs alongside the optimizer on a phone-class CPU.

Training is self-supervised. EuRoC MAV is hardware-synchronized, so shifting the IMU stream by a known amount yields exact labels. We inject offsets drawn from constant values up to ±400 ms, linear ramps, random walks and step stalls, and train the gate network with cross-entropy against the injected lag on training sequences. We evaluate on held-out sequences. Integration into VINS-Mono is minimal. The posterior's mode initializes $t_d$. Its variance becomes a prior factor on $t_d$, refreshed every window, which lets the refined offset follow non-stationary drift. When posterior entropy exceeds a threshold chosen on validation data, the $t_d$ update is frozen.

A secondary study examines the center-row approximation of Ling et al. Running the same lag search separately on horizontal bands of the image yields a row-dependent time signal. This tests directly whether a single scalar suffices under aggressive rotation. We treat this study as diagnostic, not as the main contribution.

Baselines

The do-nothing reference is VINS-Mono with temporal calibration disabled, alongside OKVIS. Online Temporal Calibration for Monocular Visual-Inertial Systems reports that uncalibrated OKVIS error spikes above 2 m under a 30 ms injected offset on EuRoC MAV. This baseline sets the floor.

The strongest existing online methods are two. The first is Qin & Shen's calibrated VINS-Mono, which reports about 0.15 m trajectory root-mean-square error (RMSE) on EuRoC MAV with a 30 ms offset, and offset RMSE below 0.7 ms in simulation. The second is Ling et al.'s random-walk model, which reports stable accuracy at 60 ms on EuRoC MAV and a 90% versus 70% initialization success rate on Galaxy S8 handheld sequences. We run both on identical injected-offset sequences, with the same feature budget and the same window length. If Ling et al.'s code is unavailable, we add their random-walk prior to VINS-Mono and label it as a reimplementation. Beating them means two things. First, matching their error within noise at 30–60 ms, where they already work. Second, keeping trajectory RMSE within a small margin of the synchronized-data error, with offset error under a few milliseconds, at 100–400 ms constant offsets and under ramp or stall drift, where the file reports they degrade.

The most dangerous baseline is the unlearned whitened cross-correlation ($G_\theta\equiv1$), using the same visual signal and the same VINS-Mono integration. A second competitor is a multi-start VINS-Mono that initializes $t_d$ on a 20 ms grid and keeps the lowest-residual hypothesis, given the same wall-clock budget. Two ablations isolate our ingredients. One restricts the lag range to ±50 ms, which isolates the global temporal view. The other swaps the phase-correlation visual signal for sparse corners, which isolates the global spatial view. For PennCOSYVIO and IndustryVI, no reported temporal-calibration result was found on Lacuna. There we report all methods ourselves under injected offsets.

Why not just…?

Why not just use hardware synchronization? Where hardware triggering exists, it should be used. The file notes, though, that consumer devices generally lack it, and that driver buffering and scheduling jitter add latency after any trigger (Modeling Varying Camera-IMU Time Offset in Optimization-Based Visual-Inertial Odometry). Our target is the regime where software timestamps are all one has.

Why not just cross-correlate gyro and visual rotation without learning anything? This is the real threat. For large constant offsets under rich motion, a plain whitened correlation may do as well as the learned version. If it does, the honest conclusion is that the global lag search is the contribution and the gate is unnecessary. We would then report exactly that. The ±50 ms ablation and the $G_\theta\equiv1$ baseline separate the two effects. We expect the gate to matter in three cases: in vibration-heavy segments, under slow or near-degenerate motion where uncalibrated correlation peaks are spurious, and in whether posterior width predicts actual offset error. We measure all three directly.

Why not just run the optimizer from many initial offsets? A grid of restarts widens the capture range. It costs one full optimization per hypothesis, however, and must be repeated whenever the offset drifts. Under an equal compute budget we test whether it matches our accuracy, and we check whether its residual-based selection can detect degenerate windows at all.

Why not just replace corner tracking with a large dense-flow network or non-local attention? A better visual front end addresses texture failure but not the temporal capture range, because the offset is still refined locally. Standard non-local blocks also carry the quadratic memory cost documented by CCNet. The sparse-versus-global visual-signal ablation measures how much of any gain comes from the image side alone.

Proposed Experiments

The decisive experiment is a sweep of injected constant offsets from 0 to ±400 ms on held-out EuRoC MAV sequences. We report offset error, trajectory RMSE and initialization success rate for every baseline. The claim fails if Qin & Shen or the multi-start baseline match us beyond 100 ms. The second experiment injects time-varying drift: ramps of several milliseconds per second, random walks, and 50–150 ms stalls. We report offset tracking error over time. This tests whether window-wise posteriors plus a random-walk prior beat Ling et al.'s random walk alone. The third experiment tests observability. We curate low-excitation segments (hover, constant velocity) and measure whether posterior entropy predicts offset error, using rank correlation and the error when updates are frozen versus allowed. The fourth tests robustness on PennCOSYVIO's reflective scenes and IndustryVI's textureless regions, using injected offsets and comparing corner-based and phase-correlation visual signals. The rolling-shutter band study runs on handheld phone recordings, with a Kalibr-style offline calibration as the constant-offset reference. It reports whether the row-wise lags deviate measurably from a single center-row value during fast rotation. The program succeeds if the global lag search extends reliable operation well past 100 ms and tracks non-stationary drift, and if the learned gate adds measurable benefit over plain correlation in at least the vibration and degeneracy regimes. If it does not, we report that instead.