End-to-end Autonomous Driving Perception with Sequential Latent Representation Learning

Jianyu Chen, Zhuo Xu, Masayoshi Tomizuka

Introduction

Traditional autonomous driving systems rely on a highly modular perception stack. In these systems, separate subsystems handle object detection, multi-object tracking, ego-vehicle localization, and environmental mapping. While this "divide and conquer" strategy is intuitive, it introduces significant challenges. Each module often requires extensive hand-tuning of parameters and heuristics, which may fail to generalize across diverse driving environments. Furthermore, maintaining synchronization and data consistency across multiple complex subsystems increases computational overhead and system fragility.

Traditional modular autonomous driving stack vs. the proposed integrated perception approach.

This paper proposes a paradigm shift: replacing the fragmented perception stack with a single, end-to-end learnable model. By leveraging sequential latent representation learning, the system compresses high-dimensional sensor data into a meaningful "hidden" state. This state is then used to simultaneously predict vehicle poses, map the environment, and track objects, effectively cleaning up the perception stack while maintaining a separate, interpretable decision-making layer for safety.

Methodology

The core of the proposed approach is a Probabilistic Graphical Model (PGM) that treats perception as a hidden state estimation problem.

Sequential Latent Representation

The model introduces a latent variable ($z_t$) that serves as a temporal summary of all historical sensor observations and vehicle actions. Unlike traditional trackers that pass explicit object lists, this model updates the latent state over time, inherently capturing the dynamics of the scene.

Variational Inference and Architecture

The system is trained by maximizing the Evidence Lower Bound (ELBO), forcing the latent space to be sufficiently informative to reconstruct multiple outputs simultaneously. The architecture utilizes CNN-based encoders for both front-view RGB camera images and Bird's-Eye View (BEV) LiDAR point clouds. These inputs are fused into the latent space, from which several specialized "heads" or decoders branch off to produce:

  1. Sensor Reconstruction: Recreating original camera and LiDAR inputs to ensure raw feature retention.
  2. Detection Masks: Predicting a BEV map of surrounding vehicles.
  3. Local Semantic Roadmap: Generating a map of lanes and road boundaries.
  4. Global Ego-Pose: Estimating the vehicle's position and heading.
The proposed end-to-end perception architecture, showing the fusion of camera and LiDAR data into a latent state $z_t$ and the subsequent multi-task decoding.

Experimental Setup

The model was developed and evaluated using the CARLA simulator, a high-fidelity open-source platform for autonomous driving research.

  • Data Collection: 50,000 environment steps were collected by navigating an ego-vehicle through a virtual town populated with 100 other autonomous vehicles to simulate dense urban traffic.
  • Evaluation: Testing was conducted over 15,000 steps starting from random points within the CARLA environment.
  • Inputs: The model processed 128x128 RGB camera images and 128x128 2D rendered LiDAR point clouds.

Key Insights and Results

The research yielded several critical insights into the benefits of integrated perception:

  • Synergy through Multi-task Learning: A primary finding was that learning road geometry (mapping) significantly improves vehicle detection. When the model was forced to reconstruct the roadmap, its detection performance reached an Average Precision (AP) of 79.4% (at a 0.1 IoU threshold). Without the roadmap task, AP dropped to 57.4%, suggesting that environmental context is vital for accurate object localization.
  • Map-less Localization: The system demonstrated the ability to estimate global pose purely from historical sensor data, achieving a global location error of approximately 7.1 meters and a heading error of 0.17 radians. While not yet at the centimeter-level precision required for some applications, this is a significant step toward localization without pre-stored HD maps.
  • Implicit Tracking: By evolving the latent state $z_t$ based on vehicle actions ($a_t$), the model performs tracking without needing explicit data association modules like Kalman Filters.

Limitations and Future Work

Despite its promise, several hurdles remain. The current use of low-resolution (128x128) inputs limits the detection of small or distant objects. The authors suggest that upgrading the relatively simple convolutional architectures to more advanced backbones, such as ResNets or Transformers, would likely yield performance gains. Furthermore, while the results in the CARLA simulator are encouraging, transitioning this "latent perception" to the real world—with its inherent sensor noise and unpredictable lighting—remains a major challenge. Future research will focus on integrating this perception model with closed-loop decision-making and planning systems.

Summary updated

Abstract

Current autonomous driving systems are composed of a perception system and a decision system. Both of them are divided into multiple subsystems built up with lots of human heuristics. An end-to-end approach might clean up the system and avoid huge efforts of human engineering, as well as obtain better performance with increasing data and computation resources. Compared to the decision system, the perception system is more suitable to be designed in an end-to-end framework, since it does not require online driving exploration. In this paper, we propose a novel end-to-end approach for autonomous driving perception. A latent space is introduced to capture all relevant features useful for perception, which is learned through sequential latent representation learning. The learned end-to-end perception model is able to solve the detection, tracking, localization and mapping problems altogether with only minimum human engineering efforts and without storing any maps online. The proposed method is evaluated in a realistic urban driving simulator, with both camera image and lidar point cloud as sensor inputs. The codes and videos of this work are available at our github repo <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">†</sup> and project website <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">‡</sup> .
BibTeX
@inproceedings{chen2020end,
  author = {Jianyu Chen and Zhuo Xu and Masayoshi Tomizuka},
  title = {{End-to-end Autonomous Driving Perception with Sequential Latent Representation Learning}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=m7u4PDersB},
}
Figure