Perception Beyond Sensors Under Uncertainty

Masha ItkinaAAAI 2021 · DOI

Introduction

Autonomous vehicles (AVs) have made significant strides in navigating controlled environments, yet urban settings present a persistent challenge: the limitation of line-of-sight sensors. When a sensor's view is obstructed by buildings, parked cars, or other large vehicles, the AV is effectively blind to potential hazards. Standard systems often default to overly cautious behavior in these "blind spots," leading to inefficient traffic flow or dangerous sudden braking.

This research outlines a framework for AVs to "see" beyond their physical sensors by predicting human behavior and inferring hidden obstacles. The core innovation lies in the explicit modeling of two distinct types of uncertainty: aleatoric uncertainty, which stems from the inherent randomness of human behavior, and epistemic uncertainty, which represents the model's own lack of knowledge regarding unfamiliar scenarios. By distinguishing between these, AVs can make more "self-aware" and safer decisions in complex urban environments.

The Problem of Uncertainty in Perception

While deep learning models excel at pattern recognition, they are notoriously poor at quantifying their own reliability. In "out-of-distribution" (OOD) scenarios—situations the model did not encounter during training—standard networks often provide high-confidence predictions that are entirely incorrect.

To navigate safely, an AV must differentiate between:

  1. Aleatoric Uncertainty: "I see the driver, but they could turn left or right."
  2. Epistemic Uncertainty: "I have never seen a vehicle move in this specific pattern before; I should not trust my prediction."

Addressing these uncertainties allows the vehicle to determine when to proceed with caution and when to hand over control or trigger a fail-safe mode.

Methodological Framework

The proposed approach integrates several advanced machine learning techniques to handle environment prediction and behavior modeling:

1. Environment Prediction via ConvLSTM

Using Convolutional Long Short-Term Memory (ConvLSTM) networks, the system treats environment prediction as a video frame prediction task. By analyzing static occupancy grid data over time, the model learns the "internal dynamics" of a scene—how objects move and interact—without requiring explicit velocity or acceleration vectors.

2. Multimodal Behavior Modeling with CVAEs

Human behavior is rarely linear. A driver approaching an intersection might turn, stop, or continue straight. Conditional Variational Autoencoders (CVAEs) are employed to model this multimodality, representing various potential paths within a latent space.

3. Efficiency through Evidential Theory

To ensure these models can run in real-time on vehicle hardware, the research applies Dempster-Shafer Evidential Theory to "sparsify" the latent spaces of CVAEs. This identifies the most representative behavioral modes, reducing computational load without losing critical diversity in predictions. This theory also facilitates multi-agent fusion, where observations from multiple visible drivers are combined to "vote" on the likely presence of hidden obstacles.

4. Higher-Order Distributions for Epistemic Awareness

To tackle epistemic uncertainty, the author proposes using higher-order distributions (such as Dirichlet distributions). By learning a distribution over distributions, the network can output a confidence level that reflects how well the current input matches its training data, providing a mathematical basis for "self-awareness."

Key Insights: Drivers as Sensors

One of the most compelling concepts in this research is the use of visible agents as proxy sensors. In a social traffic environment, the behavior of one driver is often a reaction to something another driver sees. For example, if a car in an adjacent lane suddenly brakes for no apparent reason, it serves as a strong signal that a pedestrian—hidden from the AV’s view—may be crossing the street. By modeling these social dependencies, the AV can infer the state of occluded regions.

Results and Future Work

Preliminary results demonstrate the viability of this multi-pronged approach:

  • Dynamic Learning: The ConvLSTM approach successfully predicted future states from occupancy grids, proving that explicit motion vectors are not always necessary for accurate forecasting.
  • Latent Space Optimization: The evidential sparsification method significantly reduced the sample space of CVAEs while outperforming baselines that often suffered from "mode collapse" (ignoring infrequent but critical behaviors).
  • Real-Time Mapping: Multi-agent fusion demonstrated the ability to map occluded regions in real-time using data from real-world driving experiments.

As this work progresses, the focus will shift toward improving OOD generalization. While synthetic data can help train models to recognize unfamiliarity, the infinite variety of real-world "edge cases" remains a significant hurdle. Future efforts will involve iterating on epistemic uncertainty estimation and evaluating the system against a broader range of baselines to ensure robust safety in all driving conditions.

Summary updated

BibTeX
@inproceedings{itkina2021perception,
  author = {Masha Itkina},
  title = {{Perception Beyond Sensors Under Uncertainty}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i18.17855},
}
Figure