Visuomotor Understanding for Representation Learning of Driving Scenes

Seokju Lee, Junsik Kim, Tae-Hyun Oh, Yongseop Jeong, Donggeun Yoo, Stephen Lin, In So Kweon

Abstract

Training robust models for autonomous driving typically necessitates vast quantities of human-annotated data. This paper introduces SensorFlow, a self-supervised framework that leverages readily available vehicle telemetry—such as velocity and acceleration—to learn rich representations of driving scenes. By predicting how a single image transforms based on ego-motion, the model develops an intrinsic understanding of scene geometry and semantics. SensorFlow demonstrates state-of-the-art performance in unsupervised representation learning, significantly improving downstream tasks like semantic segmentation and view synthesis.

Introduction

The development of autonomous driving systems is often bottlenecked by the requirement for massive, human-labeled datasets for tasks like semantic segmentation and depth estimation. While unlabeled video data from dashboard cameras is abundant, traditional unsupervised methods often fail to capture a holistic understanding of the environment, instead relying on local pixel-matching between consecutive frames.

The authors propose that the "free" telemetry data naturally paired with every driving video—linear velocity and angular momentum—can serve as a powerful supervisory signal. By learning the causal relationship between vehicle movement and visual changes (visuomotor understanding), a model can be forced to learn about the 3D structure of the world from a single viewpoint.

Methods

The SensorFlow architecture treats vehicle motion as a control parameter for visual prediction. The system is composed of several key components designed to bridge the gap between sensor data and visual perception.

Sensor Modulator and Flow Prediction

The Sensor Modulator acts as the core integration unit. It takes raw sensor data ($n$ units of normalized velocity and inertia) and embeds them into a latent space. These features are then fused with mid-layer activations from a visual encoder. Unlike standard optical flow methods that compare two images, SensorFlow takes a single RGB image and the corresponding sensor data to predict a dense optical flow field. This task requires the network to infer depth and scale; it must understand that distant objects move slower across the image plane than nearby ones for a given vehicle speed.

Overview of the SensorFlow architecture, showing the integration of sensor data via the Sensor Modulator to predict forward and backward optical flow from a single image.

T-Symmetry and Self-Supervision

To enhance robustness, the model employs Time Reversal Symmetry (T-symmetry). It simultaneously predicts the "future" frame ($t+1$) and the "past" frame ($t-1$). For rigid scene elements, the forward and backward motions should be consistent, allowing for a geometric regularization loss. The final training signal is derived from warping the original image using the predicted flow and comparing it to the actual adjacent frames using photometric loss and Structural Similarity (SSIM).

Experimental Results

The framework was evaluated using a custom large-scale dataset (350,000 frames across various environments) and standard benchmarks like KITTI, Cityscapes, and CamVid.

View Synthesis and Representation Learning

On the KITTI dataset, SensorFlow achieved a photometric error of 0.173, outperforming established methods like MV3D (0.241). More importantly, the representations learned by SensorFlow transferred exceptionally well to semantic segmentation. When fine-tuned on the CamVid dataset, the model achieved a 30.48 mIoU, surpassing other unsupervised approaches such as "Ego-Motion" (21.89) and "Context" (25.82).

Ablation Study

The authors conducted extensive ablation studies to determine the impact of various architectural choices and training strategies.

Ablation study results showing the reduction in photometric error and End-Point Error (EPE) as components like bidirectional motion, flow consistency, and advanced modulator structures are added.

As shown in the table above, the inclusion of bidirectional motion and flow consistency significantly reduced both photometric error and End-Point Error (EPE). The transition from a simple "stack" operation to a more complex modulator involving convolutional layers and residual blocks further refined the model's predictive accuracy.

Discussion and Conclusion

SensorFlow demonstrates that visuomotor understanding is a viable and efficient path for self-supervised representation learning in driving contexts. By forcing a network to "hallucinate" future views based on motion, it naturally acquires knowledge of 3D geometry and object semantics.

While the current model assumes a static environment—which can lead to errors around independently moving objects like other cars—the use of SSIM loss helps mitigate these effects. Future work may involve explicitly modeling dynamic agents and further refining the system's sensitivity to sensor noise. Ultimately, SensorFlow provides a scalable alternative to manual labeling, bringing unsupervised performance closer to that of fully supervised models.

Summary updated

Abstract

Dashboard cameras capture a tremendous amount of driving scene video each day. These videos are purposefully coupled with vehicle sensing data, such as from the speedometer and inertial sensors, providing an additional sensing modality for free. In this work, we leverage the large-scale unlabeled yet naturally paired data for visual representation learning in the driving scenario. A representation is learned in an end-to-end self-supervised framework for predicting dense optical flow from a single frame with paired sensing data. We postulate that success on this task requires the network to learn semantic and geometric knowledge in the ego-centric view. For example, forecasting a future view to be seen from a moving vehicle requires an understanding of scene depth, scale, and movement of objects. We demonstrate that our learned representation can benefit other tasks that require detailed scene understanding and outperforms competing unsupervised representations on semantic segmentation.
BibTeX
@inproceedings{lee2019visuomotor,
  author = {Seokju Lee and Junsik Kim and Tae-Hyun Oh and Yongseop Jeong and Donggeun Yoo and Stephen Lin and In So Kweon},
  title = {{Visuomotor Understanding for Representation Learning of Driving Scenes}},
  booktitle = {BMVC},
  year = {2019},
  url = {https://openreview.net/forum?id=AQQEOGZe2JJO},
}
Figure