A camera that runs at 20 frames per second is blind for 50 milliseconds after every shot. Event cameras are sensors in which each pixel independently reports when its brightness changes, with microsecond timing. Recent systems use them to fill this gap: they compute rich features such as segmentation or detection from the last ordinary frame, then use events to move those features forward in time. These systems move features as if every point in the scene keeps its current apparent velocity, or they learn the motion with an unconstrained network. Both choices fail in the same places. Objects accelerate, frames arrive late or get dropped, and some regions produce no events at all because they are momentarily static or textureless, so the sensor has nothing to say there.
We propose to treat the space between frames as a small physical system. Each patch of the feature map is a particle that carries its features along, has a velocity, and keeps that velocity unless something pushes on it. Events arrive whenever they arrive and correct the velocities. Inertia carries the prediction through silent periods. A small neural correction is allowed to change feature content only where transport alone cannot explain it, such as newly uncovered background or appearance change. The claim is narrow and testable. This structure should match flow-warping methods when frames are frequent, and its advantage should grow with the length of the blind interval, with object acceleration, and with event sparsity. Because the next frame enters as a measurement that updates the state rather than a reset, it should also remove the visible jumps at frame arrival.
Core Claim
The closest work splits along a single line. LiFR-Seg extracts semantic features from sparse RGB keyframes and warps them to arbitrary intermediate timestamps using event-estimated optical flow and softmax splatting. It reaches 73.82% mIoU on DSEC, within 0.09% of an ideal high-frame-rate RGB system. Its motion model is purely geometric, with no notion of inertia. Ev-3DOD fills LiDAR/camera blind time with a memoryless MLP motion field and reports discontinuous jumps when a new frame re-grounds the estimate. FusionMotion makes feature evolution continuous in time with an ODE on bird's-eye-view features and an update-predict cycle at each asynchronous measurement. Its derivative is entirely a black-box network, and it does not use event cameras.
On the other side are hybrid physics-plus-learning models, in which a known equation of motion is kept and a neural network learns only the residual it misses. Neural network augmented slider–crank models and differentiable cardiac electrophysiology correction use this design on low-dimensional, uniformly sampled systems. Egocentric event-based ping pong prediction shows that ballistic equations help event-based forecasting, but it tracks a single centroid and keeps the learned part decoupled from the physics. Receding-horizon Neural-ODE estimation from events learns continuous dynamics directly from raw events. It is black-box and limited to a synthetic Gaussian blob.
What is new here is a dense feature map whose motion between frames obeys an explicit second-order transport law. Events and frames enter as asynchronous measurements, and neural capacity is confined to the residual. Concretely, each feature-map location $i$ carries a position $p_i$, a velocity $v_i$, and a feature vector $h_i$, evolving as
$$\frac{dp_i}{dt} = v_i,\qquad \frac{dv_i}{dt} = a_\phi(\cdot),\qquad \frac{dh_i}{dt} = r_\theta(h_i, \text{local context}),$$
where $a_\phi$ is a small learned acceleration and $r_\theta$ is a learned feature residual. Both are penalized toward zero, so transport explains as much of the change as possible. The rendered feature map at a query time $t$ is obtained by splatting the particles at positions $p_i(t)$. When the learned terms are small, querying any time is a closed-form warp $p_i + v_i\tau + \tfrac12 a_i\tau^2$, where $\tau$ is the time since the last update. No iterative ODE solver is needed. This directly addresses the solver-cost objection raised against continuous-time models.
Event packets of arbitrary duration update the velocities through a learned gain, in the style of a Kalman update: the correction is weighted by how confident the event evidence is. Where no events arrive, velocity simply persists. A new keyframe updates positions and features through the same kind of gated measurement step rather than overwriting the state.
Research Program
The first component is the transport model itself. We will reuse a LiFR-Seg-style keyframe encoder and an E-RAFT-style event flow estimator, so that the only change is the state and update rule. The flow estimator's output becomes a measurement of $v$ rather than the displacement itself. Its uncertainty, analogous to LiFR-Seg's ScoreNet, sets the update gain.
The second component is the residual discipline. Following the evidence that even inexact physical priors regularize learned dynamics (Empirical PAC Bayes with partially known dynamics), we will train with a penalty on $\|r_\theta\|$ and $\|a_\phi\|$. We will also check whether the residual concentrates where it should, on disocclusion boundaries. APIC motivates a related concern: learned and physical terms can compete during optimization, so we will measure whether the residual absorbs transport if left unpenalized.
The third component is re-grounding at keyframes. We will compare a hard reset against the gated measurement update. Success means removing the jump that Ev-3DOD reports without lagging behind the true state.
The irregular, asynchronous timing of the data is treated as a constraint, not a topic. Nothing in the model assumes uniform time bins, which avoids the fixed-rate floor that event binning reintroduces (van Eden et al.). It also avoids the out-of-memory failure FusionMotion reports for dense discrete recurrence.
Calibrating per-pixel contrast thresholds, as in the receding-horizon work, is deliberately out of scope. Bundling it in would make the attribution of any gain unclear.
Baselines
The naive reference is zero-order hold: the last keyframe's prediction is reused until the next frame. Ev-3DOD reports this at 33.27 mAP on Ev-Waymo. Any method that does not clearly beat it at long blind intervals is useless.
The strongest existing segmentation competitor is LiFR-Seg, reported at 73.82% mIoU on DSEC. We will retrain it under our protocol with the same encoder, the same event representation, the same training data, and an equal hyperparameter-search budget. On full-rate DSEC, matching it within noise is the expected outcome, not a win. Beating it means a statistically significant mIoU gain in the reduced-frame-rate and event-dropout regimes.
For 3D detection, the strongest competitor is Ev-3DOD, at 48.06 mAP on Ev-Waymo. We will apply our transport state in its bird's-eye-view feature space with its detection head unchanged. There, the inertial prior corresponds to real vehicle kinematics rather than image-plane motion.
The black-box continuous competitor is a FusionMotion-style variant in which the feature derivative is an unconstrained network over event-updated features. It gets the same parameter count and compute, which isolates the value of the physical structure rather than of continuous time.
Three ablations isolate the key ingredient. First-order transport sets acceleration to zero and the velocity update gain to one, which essentially recovers flow warping in our code base. Transport without residual sets $r_\theta = 0$. Hard reset at keyframes replaces the gated update.
Why not just…?
Why not just keep flow warping, as LiFR-Seg does? On full-rate DSEC it is already within 0.09% of a high-frame-rate system, so there is little room to improve, and we do not expect to. Its weaknesses appear elsewhere. Constant-velocity splatting cannot follow curved or accelerating motion. Its confidence estimator can only down-weight regions without events; it cannot predict through them. M3ED reports E-RAFT flow at 0.934 EPE on car_urban_parking, with breakdowns under rapid rotation. Whether this matters is settled by the frame-dropping sweep: if LiFR-Seg's curve of mIoU against time-since-keyframe stays parallel to ours as intervals grow, inertia buys nothing and we will report that.
Why not just add a constant-acceleration term to the warp, with no learned dynamics or measurement updates? This is the most serious threat, because it captures much of the inertia benefit cheaply. Our first-order and no-residual ablations bracket it. A second-order warp fit by finite differences over the last few flow estimates is included as an explicit baseline. If it matches the full model, the contribution reduces to "use acceleration," and we will say so. We expect it to lose where acceleration estimates from noisy, sparse events are unreliable, which is exactly where a gated, uncertainty-weighted update should help.
Why not just learn the dynamics end to end with an unconstrained neural ODE? FusionMotion shows that this works on nuScenes, with +7.0 IoU over BEVerse. Black-box rollouts tend to drift when observations stop, however, and the matched black-box baseline is evaluated specifically in event-dropout windows to test this. If it equals our model there, the physical prior is unnecessary at this data scale.
Why not just run a Kalman filter with ballistic physics on tracked objects? For a single ball this works: the ping-pong system gains 4.8 cm over a 0.2 s horizon from physics alone, and 8.4 cm with learning added. Such a filter produces centroids, not dense semantic maps, so it cannot answer the segmentation question at all. We include it only on the ping-pong data as a sanity check of the inertial component.
Proposed Experiments
The decisive experiment is a blind-interval sweep. On DSEC, we will subsample RGB keyframes from 20 Hz down to 10, 5, and 2 Hz while keeping full event streams. We will evaluate segmentation at intermediate times and plot mIoU against time since the last keyframe for every method. Intermediate-time labels at 10 ms spacing come from SHF-DSEC; for DSEC-3DOD, the 100 FPS box annotations are used. No reported result was found on Lacuna for either SHF-DSEC or DSEC-3DOD, so all numbers there come from our own reruns. The prediction is equal performance at 20 Hz and a gap that widens as the frame rate drops.
The second check is event dropout. We will mask events in random spatial blocks for 10–50 ms windows to simulate static or textureless regions and measure how accuracy decays inside the masked regions. Flow warping has no evidence in those regions, while inertia should keep them coherent.
The third check is aggressive motion. We will use M3ED sequences from the legged Spot robot and the UAV, and Ev-Waymo for 3D detection, reporting mAP alongside a discontinuity metric: the change in prediction across each keyframe arrival compared with the change in ground truth over the same instant.
Every method will also be reported with query latency per timestamp and peak memory, so that accuracy gains are not bought with solver cost.
The idea fails cleanly under two outcomes. One is that the constant-acceleration warp matches the full model across all sweeps. The other is that the residual network, without its penalty, absorbs the transport and the penalized version underperforms.
References
- LiFR-Seg: Anytime High-Frame-Rate Segmentation via Event-Guided Propagation
- Ev-3DOD: Pushing the Temporal Boundaries of 3D Object Detection with Event Cameras
- FusionMotion: Multi-Sensor Asynchronous Fusion for Continuous Occupancy Prediction via Neural-ODE
- Neural Network Augmented Physics Models for Systems with Partially Unknown Dynamics: Application to Slider-Crank Mechanism
- Simultaneous data assimilation and cardiac electrophysiology model correction using differentiable physics and deep learning
- Egocentric Event-Based Vision for Ping Pong Ball Trajectory Prediction
- Receding-Horizon Maximum-Likelihood Estimation of Neural-ODE Dynamics and Thresholds from Event Cameras
- Learning Partially Known Stochastic Dynamics with Empirical PAC Bayes
- APIC: Orthogonalized Neuro-Symbolic Modeling for Nonlinear Dissipative Dynamics
- A comparison of visual place recognition methods using a mobile robot in an indoor environment
- M3ED: Multi-Robot, Multi-Sensor, Multi-Environment Event Dataset