Rendering Expected Looming: Map-Checked Time-to-Contact for First-Person Cameras

A camera worn on the head, or mounted on a moving robot arm, can tell how soon it will hit something without knowing any distances. It does this by watching how fast things grow in the image. The ratio of an object's image size to its rate of growth is the time-to-contact, or TTC. Fast networks now estimate this "optical expansion" from pairs of frames in a few milliseconds. However, they were built and tested on forward-driving footage, and they break when the camera rotates sharply or bounces. That is exactly how a first-person camera moves. Separately, 3D scene maps built from Gaussian splats (a scene stored as millions of small colored ellipsoids that can be drawn very quickly) give a robot or headset persistent memory of its surroundings. Robots typically use these maps by rendering full color images and matching them against the camera, which is the slow part.

We propose to use a lightweight map of the static scene to render, instead of images, the expected expansion field. This is how fast each pixel should be growing if nothing in the world were moving, given the camera's tracked motion. We then compare this cheap per-pixel prediction against the expansion actually measured in the video. Where the two agree, the map supplies a stable, low-noise TTC that survives motion blur, textureless walls, and a hand briefly covering the obstacle. Where they disagree, something unmapped or moving is present, and the measured expansion takes over. The camera's rotation, known from the same tracker, is removed exactly, because rotation's contribution to image expansion does not depend on depth. The bet is that this consistency check gives fewer false collision alarms during head turns than any purely 2D method. We also bet it does so at a latency comparable to those methods, because the map only has to draw depth at low resolution, not color.

Core Claim

Two facts make this work. The first is that the expansion produced by rotation is independent of depth. In normalized image coordinates $(x, y)$, a camera rotating with angular velocity $(\omega_x, \omega_y, \omega_z)$ produces a flow field whose divergence is

$$\operatorname{div}_{\text{rot}}(x,y) = 3\,(y\,\omega_x - x\,\omega_y),$$

which follows from differentiating the standard rotational flow equations. Sign conventions vary, but the depth-independence does not. If the angular velocity is known, the rotation-induced pseudo-looming can be subtracted everywhere, including on objects the map has never seen.

The second fact is that, for static geometry, the map predicts the translational expansion directly. Let $Z(p)$ be the depth rendered from the map at pixel $p$, and let $v_z$ be the camera's forward velocity from the tracker. The predicted inverse TTC is then

$$\hat\eta(p) = \frac{v_z}{Z(p)} .$$

Both $v_z$ and $Z$ are in the map's own arbitrary units, so the scale cancels. A map built from a single monocular camera, with no metric scale, still yields TTC in seconds. This sidesteps the scale-speed ambiguity flagged in Upgrading Optical Flow to 3D Scene Flow Through Optical Expansion.

The system computes the observed, de-rotated expansion $\eta_{\text{obs}}(p)$ with a fast 2D network. It forms the residual $r(p)=\eta_{\text{obs}}(p)-\hat\eta(p)$ and gates on it. Small residuals relative to the network's predicted uncertainty are fused by inverse variance, and the map dominates when flow is unreliable. Large residuals are labeled unmapped or moving, and $\eta_{\text{obs}}$ is trusted alone.

Relation to Prior Work

The fast 2D estimators supply the observation channel and are the main competitors. Upgrading Optical Flow to 3D Scene Flow Through Optical Expansion fits local affine transforms to flow to get dense expansion. Binary TTC recasts TTC as threshold classifications running in 6.4 ms. The Deep Scale model in TSTTC bins scale change for highway vehicles. All three use only frame pairs, with no scene memory. Binary TTC explicitly reports that rapid rotations and bumps corrupt its scaling assumption. We reuse such a network rather than replace it, and add a map-based prediction it can be checked against.

Classical work on bounding TTC from first-order flow motivates the rotation handling. Generalized Bounds for Time to Collision from First-Order Image Motion brackets TTC by combining divergence and deformation without recovering rotation, and Bounds on Translational and Angular Velocity Components gives the related velocity bounds. We instead take the rotation from map tracking and subtract it exactly. The bounds then serve as a fallback when tracking fails.

Map-based navigation constrains the design. Vision-Only Robot Navigation in a Neural Radiance World treats radiance-field density as a collision cost and tracks pose by rendering color images. That approach is slow, assumes a static scene, and ignores looming entirely. RNR-Nav avoids rendering by correlating bird's-eye-view features at 25.4 Hz, but then has no way to sense an unmapped obstacle approaching. Our design keeps rendering, but only of a single depth channel. It also uses the 2D looming signal for exactly the dynamic cases those systems miss.

On rendering cost, RTGS shows that mobile splatting latency is governed by how many screen tiles each ellipsoid overlaps. That is a warning for us: at close range, ellipsoids grow and tile overlaps rise. We adopt its efficiency-guided pruning and measure the close-range latency directly.

Finally, Egocentric Future Localization shows that cancelling head orientation, via its ground-aligned EgoRetinal map, dramatically improves first-person prediction. It does this with stereo and structure-from-motion, and it never estimates TTC. Its EgoMotion recordings give us real head trajectories.

Research Program

We will build the system in three parts.

The first part is a Gaussian-splat map of the static scene with a tracker that outputs pose and velocity at frame rate. We take the tracker off the shelf and count its time in the latency budget.

The second part is a "looming renderer." It alpha-composites only depth, at quarter resolution, skipping all color and view-dependent terms, and converts the result into $\hat\eta$. We will also try computing $\hat\eta$ analytically from each ellipsoid's projected-area rate, and compare this against differencing rendered depth between frames.

The third part is the observation network, a Binary TTC-style model fed by de-rotated expansion, extended to output a per-pixel uncertainty. It is trained on data with large rotations.

The analysis has three questions. How often does the residual gate correctly flag moving objects? How much does map fusion reduce error on blurred, textureless, and hand-occluded pixels? How does latency behave as TTC approaches zero, when splat tile overlaps grow?

Evaluation needs ground-truth TTC under real head motion with moving obstacles, and no benchmark in our file has this. We will build one in two tiers.

The semi-synthetic tier follows TSTTC's practice of using neural rendering to synthesize near-misses. We replay head trajectories recovered by structure-from-motion in the EgoMotion dataset (Egocentric Future Localization) through ScanNet scenes and insert moving objects with exact TTC. To avoid flattering our method, the test frames are rendered by a separate, higher-capacity reconstruction trained on different views, not by the map the method uses.

The real tier records a head-mounted camera with an inertial measurement unit (IMU), walking through rooms that were mapped beforehand, where people and furniture are moved after mapping. Offline depth from a co-mounted sensor provides reference TTC. We acknowledge that this reference is noisy.

We also run the observation network alone on KITTI15 and TSTTC, to confirm it matches its published numbers before any egocentric claims are made.

Baselines

The naive reference is raw image divergence converted to TTC, with no de-rotation and no map. It shows how badly head rotation contaminates the signal.

The strongest existing method is Binary TTC (Binary TTC), which reports a motion-in-depth error of 73.55 on the KITTI15 validation set at 6.4 ms on a V100. Close behind are optical expansion (Upgrading Optical Flow), with motion-in-depth error 75 on KITTI, and Deep Scale (TSTTC), with motion-in-depth error 14.4 and relative TTC error of about 6.3% for TTC under 3 s on TSTTC. These are the strongest competitors because they are the fastest and most accurate monocular TTC estimators known, and our system's observation channel is one of them. For fairness, every 2D baseline is retrained on the same egocentric training split, with the same augmentation, the same tuning budget, and the same embedded GPU, and timed end to end.

The baseline most likely to win is IMU de-rotation followed by Binary TTC. It gets the same rotation correction we get, but from the inertial sensor.

We also run a map-only baseline. It computes TTC from rendered depth and tracked velocity, in the spirit of Vision-Only Robot Navigation in a Neural Radiance World's use of the map for collision cost, and it is blind to anything unmapped.

Ablations isolate the key ingredient. One removes the residual gate and always fuses. One replaces map-tracked rotation with IMU rotation. One replaces the analytic looming renderer with depth differencing.

Beating the field means two things on the egocentric test sets. First, lower relative TTC error for TTC under 3 s on static obstacles than IMU plus Binary TTC. Second, no worse error than IMU plus Binary TTC on moving obstacles. Both must hold with a lower false-alarm rate for the "collision within τ" decision during head-turn segments, and within a stated per-frame latency budget on the same hardware.

Why not just…?

Why not just de-rotate with an IMU and run a 2D TTC network? This is the real threat. On moving obstacles with good IMU calibration it may match us, and the experiments are designed to show exactly that if it happens. The map should help where 2D expansion has nothing to measure. That includes blurred frames, textureless walls (the regime that motivated Time-to-contact from image intensity), and obstacles hidden by the wearer's hands for a few frames, which Ego4D documents as routine. The map also helps when IMU bias drifts. We report errors separately on these strata. If IMU plus Binary TTC ties us on all of them, we conclude the map is not worth its cost.

Why not just measure depth with an active sensor and differentiate it? Differencing noisy range data amplifies noise. TSTTC reports relative TTC errors above 50% for tracker- and detection-based estimators in the under-3-second regime, against about 6.3% for direct scale estimation. Active sensors also cost power that glasses and small drones lack. We include depth differencing on the real tier, where a depth sensor exists, as a reference.

Why not just use the map alone, since it already gives TTC for static surfaces? It cannot see what was not mapped. RNR-Nav and the radiance-field navigation system share this blindness to new obstacles. The moving-object stratum of our benchmark measures how often the map-only baseline misses an approaching person entirely.

Decisive Checks

Three results would settle the proposal.

The first is false collision alarms during rapid head yaw and pitch. The naive baseline should fire constantly, and both de-rotated methods should not. Our method should improve over IMU de-rotation mainly when the IMU is degraded with simulated bias.

The second is TTC error on obstacles occluded by hands and on blurred frames. This is where map fusion must win clearly, or the idea fails.

The third is latency as TTC approaches zero. If splat tile overlaps make depth-only rendering slower than the 2D network precisely at close range, as the RTGS analysis suggests is possible, we fall back to the 2D channel in that regime and report the crossover point honestly.