Discrete Latent Variables for Multimodal Trajectory Prediction

This direction is merged into another direction: multi-modal trajectory prediction. Its last version stays readable and citable here.

Version 1 of 126 papers35 conceptsfrozen 2026-10-04

When an autonomous vehicle approaches a four-way stop, it faces a fundamental challenge: the human driver across the intersection might accelerate to go first, or they might tap their brakes to yield. These are not just slightly different versions of the same path; they are distinct, mutually exclusive "modes" of behavior.

Traditional regression models often struggle here, producing a single "average" trajectory that splits the difference—resulting in a path that drives straight into the middle of the intersection at an unrealistic, half-hearted speed. To solve this, researchers have converged on a powerful architecture: the Conditional Variational Autoencoder (CVAE) with discrete latent variables.

The Multimodality Problem

In human-robot interaction (HRI), "multimodality" refers to the reality that there are multiple, equally plausible futures. As Edward Schmerling, Karen Leung, and Marco Pavone argue in Multimodal Probabilistic Model-Based Planning for Human-Robot Interaction, capturing these modes is essential for safe negotiation. If a robot only predicts the most likely outcome, it remains blind to "long-tail" risks—like a pedestrian suddenly darting across the street.

Discretizing Intent

The breakthrough in recent work is the use of discrete latent variables to represent high-level intent. Instead of a continuous "style" variable, the model is forced to choose from a set of categorical modes.

  1. Unsupervised Clustering: During training, the discrete latent space acts as an unsupervised clustering mechanism. As shown in Trajectron++ by Boris Ivanovic and colleagues, the model automatically learns to group trajectories into semantically meaningful categories like "turning left," "braking," or "accelerating."
  2. Preventing Mode Collapse: Continuous latent spaces often suffer from "posterior collapse," where the model ignores the latent variable and produces a single, blurry average. Discrete variables, often implemented via the Gumbel-Softmax trick as seen in GroupNet by Siheng Chen, force the generator to produce distinct, non-overlapping paths.
  3. Interpretability: Because each latent state corresponds to a specific behavioral mode, the model becomes "interpretable." A planner can look at the latent variable and understand why a trajectory was generated (e.g., "The model is currently simulating the 'yielding' mode").

From Prediction to Planning

Capturing these modes isn't just about better "guessing"; it’s about better "doing." In MATS: An Interpretable Trajectory Forecasting Representation for Planning and Control, Adrien Gaidon and his team demonstrate that by representing these modes as distinct dynamical systems, a robot can optimize its own path while accounting for how a human might respond to its actions.

This approach even extends to "seeing" through walls. Masha Itkina and Katherine Rose Driggs-Campbell show in Multi-Agent Variational Occlusion Inference Using People as Sensors that by observing how a human driver behaves, a discrete latent model can infer multiple hypotheses about what might be hidden in a blind spot, allowing the robot to reason about a distribution of risks rather than a single, potentially fatal assumption.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_discrete_latent_variables_for_multimodal_fd501396,
  title = {Discrete Latent Variables for Multimodal Trajectory Prediction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/discrete-latent-variables-for-multimodal-trajectory-prediction/rdr_fd5013967f324bc2b0919d1e14846812},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_discrete_latent_variables_for_multimodal_fd501396_v1,
  title = {Discrete Latent Variables for Multimodal Trajectory Prediction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/discrete-latent-variables-for-multimodal-trajectory-prediction/rdr_fd5013967f324bc2b0919d1e14846812/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}