Farther Objects Need Longer Memory: Scale-Matched Temporal Horizons for Driving Policies

Driving policies trained by imitation learning copy expert controls from camera frames. Most still look at the world through one coarse feature map from a small convolutional network, and they handle time with a recurrent cell or not at all. This proposal starts from a simple geometric fact. An object far away looks small, and when you close in on it, its image grows only slowly. Telling whether you are gaining on it therefore takes many frames. A nearby object looks large, grows quickly, and needs only a few frames. Multi-scale detectors already sort objects by image size into a feature pyramid, a stack of feature maps at progressively coarser resolution, so that small objects sit on fine maps and large objects on coarse ones. We propose to give each pyramid level its own temporal memory. Each level gets a dilated temporal convolution, a 1D filter over time that skips frames at a fixed spacing so its window grows exponentially with depth. The window is longest at the fine levels that hold distant objects and shortest at the coarse levels that hold near ones.

The core claim is narrow and falsifiable. In a camera-based imitation policy, matching each pyramid level's temporal window to the depth of the objects it represents works better than any shared window, long or short, at equal parameters and compute. The gain should show up specifically where it matters for safety: braking for distant slow or stopped vehicles at realistic speeds. If a single uniform window performs as well, the idea is wrong, and the experiments are designed to show that cleanly.

Why this specific combination

Each ingredient already exists. What is missing is a principled link between them. The strongest camera-only imitation baseline in the file, TCP, maps a single monocular frame through a ResNet-34 to waypoints and controls. It uses gated recurrent unit (GRU) modules to roll the future forward, and its ensemble reported a CARLA Leaderboard Driving Score of 75.137. TCP has no feature pyramid and no observation history. Conditional imitation learning and its real-world multi-view extension motivate the need for temporal information. In the extension, adding optical flow and extra views raised pace-car stopping success from 69% to 81%. Flow, however, compares only adjacent frames.

On the detection side, two papers establish that small objects disappear in deep, heavily downsampled layers and need their own treatment: SSD and Finding Tiny Faces. The second also argues that cues for tiny objects are qualitatively different from cues for large ones. These papers supply the pyramid but treat only static images. On the temporal side, Convolutional Sequence Modeling Revisited and MS-TCN supply the dilated temporal convolution and show it retains long context better than recurrent units. Both run on frozen, pre-extracted features with a single horizon and no control output.

The obvious combination would be to swap TCP's ResNet for a pyramid backbone and its GRU for a temporal convolutional network. That is an architecture swap, and a reviewer would rightly ask which part matters. Our contribution is a specific prediction about how temporal window length should scale across pyramid levels, together with controls that can refute it.

Core Claim and its geometric basis

Consider an object at depth $Z$ approaching the ego vehicle with closing speed $v_r$. Its image size $s$ is roughly proportional to $1/Z$. The fractional growth of $s$ per unit time is $v_r/Z$, and its inverse is the time to contact

$$\tau = \frac{s}{ds/dt} = \frac{Z}{v_r},$$

which is the quantity a braking decision depends on. Suppose size is measured with roughly fixed relative noise per frame. The image must then grow by a fixed fraction before $\tau$ can be read reliably. That fixed fraction equals $v_r T / Z$ over a window of $T$ seconds, so the required window scales as

$$T \propto \frac{Z}{v_r}.$$

Adjacent pyramid levels differ by a factor of two in the image size they are assigned. They therefore differ by roughly a factor of two in depth, and so by a factor of two in required window. Exponentially dilated convolutions double their window with each added layer of doubled dilation. The prediction follows directly: each step toward a finer pyramid level should get one additional doubling of temporal dilation.

The argument has limits, and the experiments respect them. Lateral motion measured in units of object size, which is speed times time divided by physical size, does not depend on depth. Traffic-light state changes do not either. The scale-matching argument applies to closing motion, and we predict no gain on lateral or signal-driven events.

Research Program

We build a causal, streaming policy with four parts. First, a pyramid backbone produces features at strides 8, 16 and 32. Two backbone variants test whether the effect depends on the backbone: ResNet-34 with a feature pyramid network, and Swin-T, the hierarchical windowed transformer from Swin Transformer. The ResNet variant can optionally start from PPGeo weights through the OpenDriveLab/PPGeo code. We stop at stride 8 because a buffered stride-4 map would cost too much memory.

Second, each level's channels are reduced to 64. A depthwise causal dilated temporal convolution then runs at every spatial cell. With stacked dilations 1, 2, 4, and so on, every past frame stays covered, which avoids the "holes" problem of large dilations. The schedule is 16 frames at stride 32, 32 at stride 16, and 64 at stride 8, at 10 Hz.

Third, the temporally filtered levels feed TCP's waypoint and control heads unchanged. Any difference then comes from perception and memory, not from the planner.

Fourth, ego-speed history is not fed to the temporal module. Only the current speed is supplied, as in TCP. This addresses the shortcut problem identified by PlanTF, whose planners learned to extrapolate past ego motion. Visual history can still leak ego motion through background flow, so we measure the shortcut explicitly rather than assume it away.

The streaming state per level is the buffer of past features that the temporal convolutions need. That memory cost, flagged in Convolutional Sequence Modeling Revisited, is reported directly alongside latency.

We also add a perception probe. CARLA exposes privileged labels, so for every vehicle we record depth, closing speed and true time to contact. We then fit linear probes on each level's temporally filtered features to predict time to contact, with results binned by distance. This tells us whether the matched schedule actually encodes looming better for far objects, independently of whether driving improves.

Baselines

The naive reference is TCP retrained on our data. It is single-frame, uses ResNet-34 and GRU heads, and is collected with TCP's expert across CARLA's eight towns, following the 420,000-frame protocol described in the paper. TCP is the strongest competitor because it is the best camera-only imitation method in the file, with a reported Driving Score of 75.137 for TCP-Ens on the CARLA Leaderboard. We will not compare against that leaderboard figure directly. Instead, we report our own reproduction on Town05-Long and the scenario suite below with three training seeds, and compare our model only to that reproduction. Beating TCP means a Driving Score gain larger than the seed spread, accompanied by fewer vehicle collisions.

The second baseline adds a feature pyramid to TCP but keeps it single-frame. It isolates the multi-scale contribution from the temporal one.

The key ablation uses our full architecture with three schedules of identical parameter count and identical maximum window: matched, uniform long (64 frames at every level), and inverted (longest window on the coarse level). The uniform and inverted schedules are the controls that can refute the claim. Matched counts as winning only if it beats both on braking-onset error for distant obstacles, and is not worse on aggregate Driving Score.

The fourth baseline gives every level a causal temporal attention block with a 64-frame window. Each level can then learn its own effective horizon. This tests whether hand-matched schedules are needed at all.

All models share the same data, input resolution, number of training steps, and hyperparameter search budget. Parameters and FLOPs are matched to within 5%, by widening TCP's heads where necessary.

Why not just…?

Why not just add a feature pyramid to TCP and keep it single-frame? Small-object detection improves with a pyramid, as SSD shows, but a single frame contains no looming signal at all: time to contact is a rate. The evidence from Urban Driving with Conditional Imitation Learning, where adding flow helped stopping, points the same way. The pyramid-only baseline measures this directly. If it closes most of the gap on distant-braking scenarios, the temporal component is unjustified.

Why not just use one long temporal window everywhere? This is the most serious threat, because a long window contains a short one, and a network could learn to ignore old frames at coarse levels. Our counterargument is that long windows at coarse levels add parameters and invite the history-based shortcut PlanTF describes. That is an argument, not evidence. The uniform-long ablation and the probe settle the question. If uniform long matches the matched schedule on distant braking and on the probe, we will report the scale-matching claim as refuted and recommend the simpler design.

Why not let attention learn per-level horizons? Learned horizons are a real alternative, and if they match our schedule, that is itself a positive finding. In that case we would check whether their learned attention spans grow toward fine levels as predicted. Our bet is that, from imitation data alone, attention spends its capacity unevenly and costs more at stride 8. The comparison reports both accuracy and the learned spans.

Why not just use a stronger backbone such as Swin? The file's claims about backbone dependency suggest that richer spatial context helps. But no backbone, however strong, recovers a rate from a single frame. Running the matched-versus-uniform comparison on both ResNet-FPN and Swin-T tells us whether the temporal effect survives a better backbone or is absorbed by it.

Proposed Experiments

The main evaluation uses CARLA Town05-Long plus a controlled scenario suite. In each scenario, the ego vehicle approaches a stopped or slow lead vehicle, with the vehicle first appearing at 20, 40, 80 and 120 m and with ego speed capped at 6, 10 and 14 m/s. The speed range matters. The CCTest evaluation found leading autopilots driving at 4.0–6.5 m/s, and at those speeds long horizons barely matter. Metrics are Driving Score, route completion, and infraction score, plus scenario-level collision rate and braking-onset error relative to the expert's braking distance. Following the CCTest finding that about 24% of standard-scenario accidents were caused by background traffic, collisions blamed on other vehicles are logged separately.

For data efficiency, we compare against PPGeo's reported 42% success rate with 10% of the training data. Shortcut reliance is measured with a frozen-history test: the past frames are replaced by copies of the current frame, and we record how much the outputs change in scenes where the expert accelerates from rest.

The decisive checks are four predictions. First, the matched schedule beats both uniform and inverted schedules on braking-onset error, and the margin grows with initial distance and ego speed. Second, the gain does not appear on lateral cut-ins or traffic-light events, which the argument says should be insensitive to depth. Third, probe accuracy for time to contact on far objects rises with the temporal window at fine levels, and not with the window at coarse levels. Fourth, these effects persist on Swin-T. Failing the first or third prediction refutes the core claim. Failing only the fourth would show that the effect depends on the backbone, which is still a useful result.