Human Insights Driven Latent Space for Different Driving Perspectives: A Unified Encoder for Efficient Multi-Task Inference

Huy-Dung Nguyen, Anass Bairouk, Mirjana Maras, Wei Xiao, Tsun-Hsuan Wang, Patrick Chareyre, Ramin M. Hasani, Marc Blanchon, Daniela RusarXiv 2024 · DOI · arXiv

Introduction

Autonomous driving systems must interpret complex environments through various lenses: depth perception, motion estimation, and semantic understanding. While specialized AI models often excel at individual tasks, they frequently lack the computational efficiency and shared contextual awareness required for real-time navigation. Researchers from Capgemini and MIT have addressed this by developing a unified "human-like" encoder capable of mastering seven driving-related computer vision tasks simultaneously. This approach creates a rich, multi-dimensional latent space that significantly outperforms standard ImageNet-pretrained models in downstream navigation tasks like steering angle prediction.

Problem Statement

A major challenge in autonomous driving is the "specialist" nature of current AI. While multi-task learning (MTL) offers a path toward efficiency, it is often hindered by gradient imbalance. Supervised tasks (with explicit labels, like segmentation) typically provide much stronger signals than self-supervised tasks (like depth or flow estimation), causing the model to prioritize semantics over geometry. Furthermore, generic pretraining on datasets like ImageNet fails to capture the specific spatial and temporal nuances essential for safe driving.

Methodology

The proposed architecture utilizes a Swin Tiny encoder as a shared backbone for six specialized decoders. The training process is divided into two distinct stages to ensure stability and performance.

The two-stage training pipeline: multi-task learning followed by frozen-weight steering prediction.

Technical Innovations

  1. Unified Latent Space: A single backbone extracts features for depth, pose, 3D scene flow, and three types of segmentation (semantic, instance, and panoptic).
  2. Multi-Scale Pose Decoder: To resolve ambiguities in camera motion estimation at low resolutions, the pose decoder aggregates features from multiple levels of the encoder hierarchy.
  3. Knowledge Distillation: To prevent supervised segmentation tasks from dominating the learning process, a "teacher" model (comprised of specialized encoders) guides the unified "student" encoder, ensuring a balanced representation of both geometric and semantic features.
  4. Two-Stage Training: After the initial multi-task training, the encoder weights are frozen. A small "attentive pooling" head is then added to predict steering angles, testing the robustness of the learned latent space.
Detailed architecture of the specialized decoders for depth, pose, flow, and segmentation, including the joint training logic.

Experimental Setup

The model was trained and evaluated using three primary datasets:

  • CityScapes: Provided high-quality segmentation labels and video sequences for the primary multi-task training.
  • KITTI: Used for benchmarking depth estimation and conducting ablation studies.
  • MIT Driving Dataset: Served as the evaluation ground for the downstream steering angle prediction task.

Results and Analysis

The "Frozen" Advantage

A key finding was that the frozen unified encoder outperformed its fine-tuned counterpart for steering estimation. The frozen model achieved a test error of 5.41, compared to 16.46 for the unfrozen version. This suggests that multi-task training creates a latent space so robust that further fine-tuning for a specific task may actually degrade the general context necessary for navigation.

Domain-Specific Superiority

The unified encoder crushed the standard ImageNet-pretrained encoder (test error of 11.03) on steering tasks. This confirms that driving-specific visual features are fundamentally different from general object features found in standard datasets.

Competitive Performance and Efficiency

Despite the complexity of handling seven tasks at once, the model achieved performance on par with specialized state-of-the-art (SOTA) models like OneFormer (segmentation) and DynamoDepth (depth). By consolidating these tasks into a single encoder, the system significantly reduces the computational footprint and inference time required for autonomous operation.

Conclusion and Future Work

This research demonstrates that a unified, multi-task latent space provides a superior foundation for autonomous driving compared to specialized or generic models. While the model generalizes well, the authors noted that training error for steering remains a target for optimization. Future work will focus on refining prediction heads for high-precision control (e.g., braking) and exploring the interpretability of the dense latent space to better understand the decision-making process of self-driving systems.

Summary updated

Abstract

Autonomous driving systems require a comprehensive understanding of the environment, achieved by extracting visual features essential for perception, planning, and control. However, models trained solely on single-task objectives or generic datasets often lack the contextual information needed for robust performance in complex driving scenarios. In this work, we propose a unified encoder trained on multiple computer vision tasks crucial for urban driving, including depth, pose, and 3D scene flow estimation, as well as semantic, instance, panoptic, and motion segmentation. By integrating these diverse visual cues-similar to human perceptual mechanisms-the encoder captures rich features that enhance navigation-related predictions. We evaluate the model on steering estimation as a downstream task, leveraging its dense latent space. To ensure efficient multi-task learning, we introduce a multi-scale feature network for pose estimation and apply knowledge distillation from a multi-backbone teacher model. Our findings highlight two key findings: (1) the unified encoder achieves competitive performance across all visual perception tasks, demonstrating strong generalization capabilities; and (2) for steering estimation, the frozen unified encoder-leveraging dense latent representations-outperforms both its fine-tuned counterpart and the same frozen model pretrained on generic datasets like ImageNet. These results underline the significance of task-specific visual features and demonstrate the promise of multi-task learning in advancing autonomous driving systems. More details and the pretrained model are available at https://hi-computervision.github.io/uni-encoder/.
BibTeX
@misc{nguyen2024human,
  author = {Huy-Dung Nguyen and Anass Bairouk and Mirjana Maras and Wei Xiao and Tsun-Hsuan Wang and Patrick Chareyre and Ramin M. Hasani and Marc Blanchon and Daniela Rus},
  title = {{Human Insights Driven Latent Space for Different Driving Perspectives: A Unified Encoder for Efficient Multi-Task Inference}},
  year = {2024},
  eprint = {2409.10095},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.10095},
}
Figure