Abstract
Training robust models for autonomous driving typically necessitates vast quantities of human-annotated data. This paper introduces SensorFlow, a self-supervised framework that leverages readily available vehicle telemetry—such as velocity and acceleration—to learn rich representations of driving scenes. By predicting how a single image transforms based on ego-motion, the model develops an intrinsic understanding of scene geometry and semantics. SensorFlow demonstrates state-of-the-art performance in unsupervised representation learning, significantly improving downstream tasks like semantic segmentation and view synthesis.
Introduction
The development of autonomous driving systems is often bottlenecked by the requirement for massive, human-labeled datasets for tasks like semantic segmentation and depth estimation. While unlabeled video data from dashboard cameras is abundant, traditional unsupervised methods often fail to capture a holistic understanding of the environment, instead relying on local pixel-matching between consecutive frames.
The authors propose that the "free" telemetry data naturally paired with every driving video—linear velocity and angular momentum—can serve as a powerful supervisory signal. By learning the causal relationship between vehicle movement and visual changes (visuomotor understanding), a model can be forced to learn about the 3D structure of the world from a single viewpoint.
Methods
The SensorFlow architecture treats vehicle motion as a control parameter for visual prediction. The system is composed of several key components designed to bridge the gap between sensor data and visual perception.
Sensor Modulator and Flow Prediction
The Sensor Modulator acts as the core integration unit. It takes raw sensor data ($n$ units of normalized velocity and inertia) and embeds them into a latent space. These features are then fused with mid-layer activations from a visual encoder. Unlike standard optical flow methods that compare two images, SensorFlow takes a single RGB image and the corresponding sensor data to predict a dense optical flow field. This task requires the network to infer depth and scale; it must understand that distant objects move slower across the image plane than nearby ones for a given vehicle speed.
T-Symmetry and Self-Supervision
To enhance robustness, the model employs Time Reversal Symmetry (T-symmetry). It simultaneously predicts the "future" frame ($t+1$) and the "past" frame ($t-1$). For rigid scene elements, the forward and backward motions should be consistent, allowing for a geometric regularization loss. The final training signal is derived from warping the original image using the predicted flow and comparing it to the actual adjacent frames using photometric loss and Structural Similarity (SSIM).
Experimental Results
The framework was evaluated using a custom large-scale dataset (350,000 frames across various environments) and standard benchmarks like KITTI, Cityscapes, and CamVid.
View Synthesis and Representation Learning
On the KITTI dataset, SensorFlow achieved a photometric error of 0.173, outperforming established methods like MV3D (0.241). More importantly, the representations learned by SensorFlow transferred exceptionally well to semantic segmentation. When fine-tuned on the CamVid dataset, the model achieved a 30.48 mIoU, surpassing other unsupervised approaches such as "Ego-Motion" (21.89) and "Context" (25.82).
Ablation Study
The authors conducted extensive ablation studies to determine the impact of various architectural choices and training strategies.
As shown in the table above, the inclusion of bidirectional motion and flow consistency significantly reduced both photometric error and End-Point Error (EPE). The transition from a simple "stack" operation to a more complex modulator involving convolutional layers and residual blocks further refined the model's predictive accuracy.
Discussion and Conclusion
SensorFlow demonstrates that visuomotor understanding is a viable and efficient path for self-supervised representation learning in driving contexts. By forcing a network to "hallucinate" future views based on motion, it naturally acquires knowledge of 3D geometry and object semantics.
While the current model assumes a static environment—which can lead to errors around independently moving objects like other cars—the use of SSIM loss helps mitigate these effects. Future work may involve explicitly modeling dynamic agents and further refining the system's sensitivity to sensor noise. Ultimately, SensorFlow provides a scalable alternative to manual labeling, bringing unsupervised performance closer to that of fully supervised models.
Summary updated