Space-Time Social Volumes for Highway Trajectory Prediction

Predicting where nearby cars will be in five seconds depends mostly on how they move relative to each other. A car closing the gap to the one ahead, a neighbor drifting toward the lane line, or a hole opening in the next lane are all events that unfold jointly in space and time. Yet the standard interaction-aware predictors summarize each car's history separately with a recurrent network, which is a model that reads a sequence one step at a time. They place these summaries into a small spatial grid around the target car, and only then mix information across cars with 2D convolutions. The grid therefore records where each neighbor ended up, not how neighbors moved through the grid relative to one another.

We propose to build the neighborhood as a three-dimensional volume of time × longitudinal bins × lanes. Each cell stores the kinematic state of whatever vehicle occupies it at that instant. The volume is processed with 3D convolutions, filters that slide over space and time together, as in video models. In plain terms, we treat the last three seconds of a highway neighborhood as a tiny video and let the network detect relative-motion patterns directly. The question we want to settle is whether interaction is better captured by doing temporal processing on the spatial grid before pooling, rather than per car before pooling. We also want to know whether any gain is concentrated where the hypothesis says it should be: in scenes with large relative motion.

Core Claim

The claim is that factorized predictors lose interaction information at a specific point. Each per-agent recurrent encoder compresses one vehicle's track in isolation, so the relative timing between neighbors must be reconstructed later from static summaries. Two examples are Convolutional Social Pooling for Vehicle Trajectory Prediction, which places per-vehicle LSTM states into a 13×3 "social tensor" and applies 2D convolutions, and Interaction-Aware Trajectory Prediction of Connected Vehicles using CNN-LSTM Networks, which does the same over a fixed 3×3 grid of exactly eight neighbors. In a grid anchored to the target vehicle, relative motion appears as occupancy and velocity patterns moving across cells over time. That is exactly the kind of signal 3D filters detect in video, according to Learning Spatiotemporal Features with 3D Convolutional Networks, which showed that homogeneous 3×3×3 kernels capture motion without precomputed optical flow.

We make three predictions. First, a grid-first spatiotemporal encoder should reduce long-horizon error relative to the per-agent-first pipeline at matched capacity. Second, the reduction should be concentrated in high-relative-motion scenes. Third, it should vanish when neighbor motion histories are removed from the volume.

This regime also sidesteps the main known weakness of 3D convolutions. Their memory cost grows quickly with volume size, which limited C3D and Deep End2End Voxel2Voxel Prediction to 16-frame clips. A highway social volume, however, is about 30 × 13 × 3 cells. At that size, full-length history fits easily and the cost is negligible.

The genuine novelty is narrow, and we state it that way. The research file contains no trajectory predictor that convolves jointly over time and the inter-agent grid. All grid-based predictors in the file apply a temporal encoder per agent first:

Occupancy-grid forecasters work on dense LiDAR maps rather than tracked-agent interaction grids, and they use different temporal machinery:

Research Program

Input volume. For each target vehicle, we build a volume with T = 30 frames (3 s at 10 Hz), 13 longitudinal bins, and 3 lanes, matching the CS-LSTM grid so that representation is not a confound. The grid is anchored to the target vehicle's position at each frame. Each cell carries five channels:

  • an occupancy flag;
  • longitudinal and lateral velocity relative to the target;
  • the sub-cell offset of the occupant;
  • vehicle length.

The target's own track is encoded separately, exactly as in CS-LSTM. Because the grid holds whatever vehicle occupies each cell, the number of neighbors is unconstrained, which removes the fixed-eight-neighbor assumption of the CNN-LSTM model.

Encoder and decoder. The encoder is a small stack of 3×3×3 convolutions that downsample the time axis progressively and end with pooling into a context vector. To isolate the encoder, we reuse the CS-LSTM maneuver-based decoder unchanged. It has six maneuver classes, combining lateral left / right / keep with longitudinal normal / braking, and outputs bivariate Gaussian position distributions. This keeps the multimodality mechanism fixed, so any change is attributable to how the past is encoded.

Variants. We build a ladder of encoders that differ in one ingredient each:

  • full 3D convolution on the grid;
  • (2+1)D convolution on the grid, meaning temporal convolution per cell followed by spatial convolution;
  • 2D convolution with time stacked into channels;
  • the original per-agent LSTM followed by 2D pooling.

We also run a "frozen-neighbors" variant of the 3D model. It replaces every neighbor's history with its last observed state, which removes relative-motion information while preserving the final configuration.

Baselines

Naive reference. The do-nothing reference is a constant-velocity extrapolation of the target vehicle. Alongside it we run a vanilla per-vehicle LSTM. CNN-LSTM reports 6.9017 m RMSE at 5 s for a vanilla LSTM on NGSIM US-101 under its own protocol. These references bound how much any interaction modeling is worth on our split.

Primary baseline: CS-LSTM. The primary competitor is Convolutional Social Pooling, which reports 5 s RMSE of 4.37 m on NGSIM US-101/I-80, against 4.55 m for Social LSTM, and an NLL of 4.22 with its maneuver decoder. It is the strongest competitor on this question because it uses the same grid and the same decoder, and differs from our model only in where temporal processing happens.

Strongest reported method. Attention Based Vehicle Trajectory Prediction (MHA-LSTM) reports about a 10% error reduction over CS-LSTM on NGSIM and highD; no absolute RMSE for it was found in the file. On highD, Relational Recurrent Neural Networks For Vehicle Trajectory Prediction (L-RRNN) reports 3.34 m longitudinal and 0.48 m lateral RMSE at 5 s. We reimplement both inside our pipeline.

We do not treat the CNN-LSTM figure of 2.272 m as a target. It uses a different horizon segmentation and a fixed neighbor set, so it is not comparable to the CS-LSTM protocol.

Fairness protocol. All models are trained on identical NGSIM and highD splits, using the published CS-LSTM split for NGSIM. Encoder parameter counts are matched within 10%. Every model receives the same hyperparameter search budget of 20 trials and is evaluated over three seeds.

What counts as beating CS-LSTM. Our CS-LSTM reimplementation must first reproduce roughly 4.37 m. The 3D model must then lower 5 s RMSE and NLL beyond seed variance, and also beat the strongest simple alternative described below. The frozen-neighbors ablation is the test of mechanism: if the 3D model's advantage survives it, the gain is not coming from relative motion and the core claim fails.

Why not just…?

Why not just feed each neighbor's LSTM its trajectory relative to the target car? This is the most serious threat. Per-agent encoders given relative coordinates and velocities can represent gap closing for each neighbor individually. MHA-LSTM(+f) already gains by adding dynamics features. What they still cannot represent before pooling is the joint timing between two neighbors, for example a car in the adjacent lane slowing just as a gap ahead opens. We include a relative-input CS-LSTM as a baseline under the same budget. If it matches the 3D model, the claim reduces to "use relative inputs," and we will report that outcome.

Why not just stack time into the channels of a 2D CNN? This approach is cheaper and also operates grid-first. The video evidence is suggestive but comes from another domain: Voxel2Voxel reports 76.0% versus 55.7% for 2D baselines on GATECH segmentation. On a grid this small, the first layer of a channel-stacked network might lose little information. We run this variant directly rather than assume the outcome.

Why not just use attention over agents, which handles non-local interactions? MHA-LSTM and L-RRNN argue that grid locality is too rigid. On a 13×3 highway grid, however, a few 3D layers already cover the whole neighborhood, so locality should matter less than when information is mixed. If attention wins, we add the obvious hybrid, 3D convolution over time followed by attention over cells, and test whether grid-first temporal processing still helps inside an attention model.

Why not just use (2+1)D convolutions? If the factorized grid-first variant matches full 3D convolution, we will recommend it as the simpler model. The core claim concerns temporal processing on the grid before pooling, not 3×3×3 kernels specifically.

Proposed Experiments

We train all variants on NGSIM and highD and report RMSE at horizons of 1 to 5 s together with NLL, plus lateral and longitudinal RMSE on highD so results are comparable with L-RRNN.

The decisive check is a stratified evaluation. We split test segments by a relative-motion score: the maximum absolute relative longitudinal velocity of any neighbor within two cells, together with any lane change into or out of the target's neighborhood. The hypothesis predicts a 3D-over-CS-LSTM gain that is clearly larger in the top tercile than in steady car-following, and that collapses under the frozen-neighbors ablation. A uniform gain across strata, or a gain that survives freezing, would mean the improvement comes from capacity or optimization rather than relative motion. In that case we would reject the mechanistic claim even if average RMSE improves.

As a secondary check, we test generalization to unseen grid configurations, following the zero-overlap experiment in CS-LSTM, to confirm that the 3D encoder retains convolutional translation invariance. Finally, we measure encoder latency against CS-LSTM to confirm that the extra computation is negligible at this volume size.