Discrete Classification for Multi-Modal Trajectory Forecasting

This direction is merged into another direction: discrete trajectory classification. Its last version stays readable and citable here.

Version 1 of 117 papers24 conceptsfrozen 2026-10-04

When predicting where a pedestrian will walk or how a car will turn, the most intuitive approach seems to be regression: training a model to output the exact $(x, y)$ coordinates of the next position. However, this approach suffers from a fundamental flaw known as the "average path" problem. If a person reaches a fork in the road and can go either left or right, a regression model—minimizing mean squared error—will often predict the average of those two paths: straight into the grass.

To solve this, a growing body of research suggests we should stop treating motion as a continuous coordinate problem and start treating it as a discrete classification task.

The Multi-Modality Problem

The core challenge in motion forecasting is multi-modality. In the real world, agents make distinct, mutually exclusive choices. Mathieu Salzmann and others have noted that traditional regression models struggle to represent these "either/or" scenarios because they are typically constrained to output a single deterministic value or a simple unimodal distribution (like a Gaussian).

By discretizing the output space, we allow the model to assign probabilities to multiple distinct outcomes. As Francesco Giuliari, Irtiza Hasan, and their colleagues demonstrate in Transformer Networks for Trajectory Forecasting, mapping continuous coordinates into discrete spatial bins transforms forecasting into a classification problem. This allows a standard softmax layer to represent a complex, multi-modal probability distribution where the model can say, "There is a 50% chance they go left and a 50% chance they go right," without ever suggesting they go straight.

From Coordinates to Tokens

If motion is a classification task, what exactly are the classes? Researchers have proposed several ways to "tokenize" the continuous world:

  1. Spatial Bins: Dividing the environment into a grid. Anh Totti Nguyen and Michael A. Alcorn used this in baller2vec to model the complex interactions of basketball players, finding that discretization allowed the model to capture non-overlapping paths that LSTMs missed.
  2. Velocity Clusters: Instead of where the agent is, we can discretize how they move. In ArTIST, motion is represented as a set of learned "motion primitives" or velocity clusters. This is particularly useful for tracking objects through occlusions, as the model can maintain multiple hypotheses about an object's velocity.
  3. Motion Templates: Modern approaches like Trajeglish by Jonah Philion and Sanja Fidler treat traffic as a language. They discretize trajectories into a "vocabulary" of motion tokens, allowing them to use GPT-style architectures for next-token prediction.

Why Discretization Works

Beyond handling multi-modality, discretization offers several structural advantages:

  • Training Stability: Zelun Luo and Li Fei-Fei found in their work on Unsupervised Learning of Long-Term Motion Dynamics that predicting quantized "atomic flows" is significantly more robust than regressing exact coordinates, which can be noisy and prone to divergence over long horizons.
  • Interpretability: When motion is mapped to categorical bins, we can apply classification-based interpretability tools. Omer Sahin Tas and Royden Wagner show in Words in Motion that these discrete bins can represent specific physical behaviors—like "hard braking" or "sharp left"—making the model's "thoughts" more legible to humans.
  • Scaling: As seen in SMART by Wei Wu and colleagues, treating motion as next-token prediction allows trajectory models to inherit the power-law scaling laws observed in Large Language Models.

Efficiency and Probabilistic Scenarios

While generative models like Diffusion or GANs can also handle multi-modality, they are often computationally expensive because they require many random samples to map out the distribution of possible futures.

A more efficient alternative is the "Probabilistic Scenarios" paradigm proposed by Xilin Dai, Qiang Xu, and others in From Samples to Scenarios. Instead of sampling, their model, TimePrism, directly outputs a finite set of discrete trajectories and their associated likelihoods in a single forward pass. This approach provides the same multi-modal benefits as discretization but with massive gains in speed—sometimes up to five orders of magnitude faster than generative sampling.

By moving from "where is the point?" to "which scenario is most likely?", these researchers are building motion models that are not only more accurate but also more aligned with the discrete decision-making processes of the real world.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_discrete_classification_for_multi_modal__2196ab29,
  title = {Discrete Classification for Multi-Modal Trajectory Forecasting},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/discrete-classification-for-multi-modal-trajectory-forecasting/rdr_2196ab2907ff43eab0a0e03a17c490b9},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_discrete_classification_for_multi_modal__2196ab29_v1,
  title = {Discrete Classification for Multi-Modal Trajectory Forecasting},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/discrete-classification-for-multi-modal-trajectory-forecasting/rdr_2196ab2907ff43eab0a0e03a17c490b9/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}