Driving systems fuse cameras, LiDAR and radar under two assumptions. The sensors are synchronized to a common clock, and every sensor is present. Neither holds reliably on the road. This proposal tests a different way to train such a system. Place every sensor frame into one stream in the order it arrived, whatever its source and whatever the gaps. Then supervise the model only with the ordered list of events that happened, such as "vehicle ahead brakes, stops, turns left", with no timestamps. The loss is connectionist temporal classification (CTC). CTC is a sequence loss that sums over every possible way of assigning the stream's frames to the target events, so the model never needs to be told which frame shows which event. Nothing in this setup assumes a fixed number of frames or a fixed sensor clock. A dropped LiDAR sweep, a delayed camera frame or a lost radar just changes the length and makeup of the stream. The model is asked to recover the same event list regardless.
The second ingredient addresses a known failure of fusion systems. A broken sensor often produces confident wrong answers, which Resilient Sensor Fusion Under Adverse Sensor Failures via Multi-Modal Expert Fusion calls the calibration trap. We borrow the idea from Variational Connectionist Temporal Classification of splitting CTC's per-frame decision into two parts: "is there an event boundary here?" and "which event is it?". We condition the first part on which sensor produced the frame. Frames from a sensor carrying no usable evidence should then learn to say "nothing here" rather than guess confidently. In two sentences: alignment-free sequence supervision over the raw, interleaved sensor stream should make event recognition robust to dropout and delay by construction. Splitting the existence decision by sensor should make its confidence trustworthy when a sensor fails.
Core Claim
The claim has three parts. First, a model trained with CTC on the timestamp-ordered, unsynchronized multi-sensor stream will lose less event-recognition accuracy under sensor dropout and clock or transmission delay than models that synchronize first. Second, it will also lose less than models trained with frame-level labels interpolated to each sensor's timestamps. Third, it will match them on clean, synchronized data.
The factorized, sensor-conditioned existence decision should additionally keep confidence discriminative under failure. We will measure this as recall at a fixed high precision, the metric under which Var-CTC reported its gains on scene text.
The new element is not CTC, and not robustness to modality dropout. It is treating the physical sensor interleaving itself as the "time axis" that CTC sums over. This makes the question of asynchrony a matter of how the loss handles alignment, instead of something a separate module must predict away.
What Each Line of Work Contributes
CTC supplies the mechanism. It extends the label set with a blank symbol and emits one symbol per input frame. A fixed collapse rule then merges repeats and deletes blanks, and dynamic programming sums the probability of every frame-level path that collapses to the target. Its known weakness constrains our design. Var-CTC shows that blanks dominate training, producing narrow, badly placed spikes and uncalibrated confidence. Var-CTC fixes this by separating the blank-versus-symbol decision from symbol identity, but only on single-image text recognition.
We reuse that factorization and change what conditions it. At stream position $t$, with feature $h_t$ from a sensor of type $m_t$ (camera, LiDAR or radar), the model predicts
$$p(\pi_t = \epsilon \mid h_t) = 1 - \sigma\big(g_{m_t}(h_t)\big), \qquad p(\pi_t = c \mid h_t) = \sigma\big(g_{m_t}(h_t)\big)\,\mathrm{softmax}_c\big(f(h_t)\big),$$
where $\pi_t$ is the symbol emitted at position $t$, $\epsilon$ is the blank, $c$ ranges over event classes, and $\sigma$ is the logistic function. $g_{m_t}$ is a per-sensor "evidence present" head, and $f$ is a shared event classifier. The existence head is per sensor because the physics differs by sensor: radar is sparse, and cameras fail at night. The classifier is shared because the events are the same regardless of which sensor sees them. We will train either with plain CTC likelihood over this factorized distribution or with Var-CTC's label-aware variational bound, and treat that choice as an ablation.
A known hard failure of CTC is that the loss is undefined when the stream is shorter than the target list. We handle this with a labeling rule, not a loss hack. An event is kept in the target only if at least one surviving frame falls within its time support. This rule is computable from recorded timestamps and makes the target reflect what was actually observable.
Multi-task learning enters in two places. The factorization above is a two-part decision inside one loss. In deployment, the event head would also share a backbone with 3D detection. That joint training can hurt detection, for the two reasons RotoGrad: Gradient Homogenization in Multi-Task Learning identifies: one task's gradients can be much larger than the other's, or the two can point in opposing directions. RotoGrad corrects both by rescaling gradient magnitudes and learning task-specific rotations of the shared features. It has only been tested on static images. We will measure how aligned the CTC and detection gradients are before reaching for it. We apply it only if we observe conflict, so that it serves as a tool rather than a claim.
Sensor fusion supplies the regime and the adversaries. MoME handles failure by routing each object query to a LiDAR, camera or fused expert before prediction. Routing before prediction sidesteps the calibration trap, but MoME works frame by frame on synchronized input. BEVSync handles delay by predicting where delayed infrastructure features would be now. It degrades once delays exceed 700 ms, which motivates an approach that does not extrapolate. MSC-Bench supplies the corruption taxonomy, including temporal synchronization errors, single-sensor and concurrent failures, and weather. It also shows that fusion degrades sharply under concurrent corruption. PETRv2 aligns frames through ego-pose transforms and assumes a fixed, synchronized frame rate. That is exactly the assumption we drop.
Two related works mark the boundary. Point2Seq casts each 3D box as an ordered sequence of attributes decoded one at a time from LiDAR alone. METDrive regresses waypoints with a temporal consistency loss. Neither sums over alignments between observations and targets.
Research Program
The testbed must isolate temporal alignment from detection quality. We therefore build the event task on nuScenes. Its 20-second scenes carry 6 cameras, one LiDAR and 5 radars running at different native rates, with raw per-sensor timestamps. Annotations exist only at 2 Hz keyframes.
We derive two event-sequence tasks from the annotated trajectories. The first is agent behavior. For each tracked vehicle or pedestrian, we derive an ordered list of maneuver events from its box track: start moving, stop, turn left, turn right, lane change, cross. The model sees sensor features pooled inside that agent's ground-truth box for every raw sweep or frame, so detection errors do not confound the result. The second is ego maneuvers, predicted from exteroceptive sensors only, with IMU and vehicle-bus signals withheld. These event types mirror the Ego and Agent interaction categories of STSBench, whose STSnu multiple-choice set we use as a sanity check on the label taxonomy, not as a training target.
The encoder is a per-sensor feature extractor. Each frame's token carries a learned sensor-type embedding and a relative time encoding. A temporal transformer runs over the interleaved stream, followed by the factorized CTC head.
Corruption is applied at the stream level, following MSC-Bench's categories. We apply per-sensor dropout rates, whole-sensor failure, and clock offsets of plus or minus 50 ms (the range BEVSync studies). We also apply transmission delays up to and beyond 700 ms, and weather corruptions from the nuScenes-C protocol used by MoME.
In the second phase, we attach the event head to a shared-backbone 3D detector on nuScenes. We ask whether joint training preserves detection accuracy and whether detection robustness under dropout improves. We log per-step gradient cosine similarity and norm ratio between the CTC and detection losses, and apply RotoGrad-style balancing only if conflict appears. A stretch goal ports the method to DAIR-V2X, where infrastructure frames arrive late for real, and compares against BEVSync's protocol.
Baselines
The naive reference point is a frame-level classifier with synchronization. We resample all sensors onto the 2 Hz keyframe clock by nearest-timestamp matching. We train a per-keyframe event classifier with cross-entropy on labels read off the annotated tracks, then decode by merging repeats. This is what one would build without thinking about alignment. It must match the proposal on clean data, or the event task is badly posed.
The strongest simple competitor is the same frame-level classifier run on the interleaved stream, with modality-dropout augmentation during training. Each sensor frame gets a label interpolated from the nearest keyframe annotation. It sees exactly our tokens, our encoder and our corruption schedule. The only difference is timestamp-assigned per-frame labels instead of CTC's sum over alignments. This is the comparison that tests the core claim.
The strongest synchronize-first competitor uses a learned compensator in the spirit of BEVSync. It predicts each delayed sensor's features at the reference time, then applies vanilla CTC on the resulting synchronized clock. On DAIR-V2X under 300 ms delay, BEVSync reports 28.33 mAP@3D at IoU 0.3 and 13.80 at IoU 0.5. Late fusion with a Kalman filter reaches 18.67 and 10.12 (BEVSync). Those numbers are for detection, not for our event task. For event-sequence error rate on nuScenes, there is no reported result found on Lacuna, so every event-task baseline is trained by us.
The robustness competitor is a MoME-style router. It uses per-sensor and fused temporal encoders, with a learned router trained under simulated dropout and a softmax CTC head on top. On nuScenes under LiDAR drop, MoME reports that a single-decoder baseline falls from 72.9 to 44.7 NDS, while MoME improves mAP by 4.2 points over prior methods (MoME). For the detection phase, MoME's own reported figures are the reference, and our detector must stay within noise of its clean NDS.
The ablations that isolate the key ingredients are four. We compare plain softmax CTC on the interleaved stream against the factorized head. We compare a shared existence head against per-sensor heads. We compare Var-CTC's variational bound against plain likelihood. For the detection phase, we compare joint training with and without gradient balancing.
For fairness, all models share the same per-sensor encoders, the same token budget, the same corruption schedule during training, the same number of training steps, and the same hyperparameter search of equal size (learning rate, dropout rate, loss weights) on a held-out split of nuScenes training scenes. Beating a baseline means lower event error rate under corruption, averaged over three seeds with non-overlapping intervals. The error rate is edit distance divided by target length. In addition, we require no more than one point of degradation on clean data.
Why not just…?
Why not just train a frame-level classifier on interpolated labels with modality dropout? This is the real threat. On nuScenes, keyframe annotations do give approximate timestamps, so alignment-free training may buy nothing when labels are dense and sensors are near-synchronous. Our expectation is that the gap opens only under asynchrony and delay. There, a timestamp-assigned label attaches to a frame whose content reflects a different moment. CTC lets the model place the event wherever the evidence actually appears in the stream. The strongest-competitor comparison above settles this directly, broken out by delay magnitude.
If the frame-level model matches ours at every delay, the claim fails. The honest fallback would then be the weak-label regime: supervision given only as event lists, for example from driving logs or language descriptions, where frame-level labels do not exist. We will run that regime too, by withholding keyframe timing from all models.
Why not synchronize first with a Kalman filter or a learned compensator? Kalman extrapolation already trails learned alignment badly on DAIR-V2X: 18.67 versus 28.33 mAP@3D at 300 ms. BEVSync itself degrades past 700 ms because extrapolation compounds error. Our method does not extrapolate, so its degradation with delay should be flatter. The delay-sweep experiment measures the slope of both curves. If the compensator wins at short delays and loses at long ones, that is a useful and publishable crossover point, not a failure.
Why not route to per-sensor experts as MoME does? MoME routing works, but it is frame-synchronous and adds an expert per sensor combination. Our per-sensor existence head is a much lighter version of the same idea inside the loss. If a MoME-style temporal router beats our factorized head on recall at 98% precision under sensor failure, the factorization is not the right place to handle failure. We would report that outcome.
Why not plain CTC on the interleaved stream? Var-CTC's evidence says plain CTC concentrates on blanks and gives confidence that cannot separate right from wrong. At 98% precision, Var-CTC raised recall by 31.7% to 50.8% on scene text (Variational Connectionist Temporal Classification). Whether this carries over to sensor streams is untested. The softmax-CTC ablation measures it directly.
Decisive Experiments
The first check is a delay-and-dropout sweep on the nuScenes agent-behavior task. We plot event error rate for all models as LiDAR, camera and radar dropout rates rise independently and jointly. We also plot it as clock offsets and transmission delays grow from 0 to beyond 700 ms. We summarize with MSC-Bench-style resilience scores, the ratio of corrupted to clean performance. The claim holds if our method has the flattest curve with equal clean accuracy, and fails if the interpolated-label frame-level model matches it.
The second check is calibration under failure. With one sensor zeroed or replaced by MSC-Bench corruptions, we report recall at 98% precision and expected calibration error for the event outputs. We also report the mean existence probability per sensor on corrupted frames. That value should fall toward blank for the failed sensor and stay flat for the healthy ones. If it does not, the factorization is not doing what we claim.
The third check is the multi-task phase on nuScenes detection: clean NDS and mAP, and NDS under LiDAR drop, compared with MoME's reported degradation. We also report gradient cosine and norm-ratio traces between the CTC and detection losses. The outcome decides whether gradient balancing is needed at all, and whether the event objective helps or hurts the detector it shares a backbone with.
References
- Resilient Sensor Fusion Under Adverse Sensor Failures via Multi-Modal Expert Fusion
- Variational Connectionist Temporal Classification
- RotoGrad: Gradient Homogenization in Multi-Task Learning
- BEVSync: Asynchronous Data Alignment for Camera-based Vehicle-Infrastructure Cooperative Perception Under Uncertain Delays
- MSC-Bench: Benchmarking and Analyzing Multi-Sensor Corruption for Driving Perception
- PETRv2: A Unified Framework for 3D Perception from Multi-Camera Images
- Point2Seq: Detecting 3D Objects as Sequences
- METDrive: Multi-modal End-to-end Autonomous Driving with Temporal Guidance
- nuScenes: A multimodal dataset for autonomous driving
- STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Driving