Adaptive Visual Interaction Based Multi-Target Future State Prediction For Autonomous Driving Vehicles

Li Du, Zixuan Wang, Leiquan Wang, Zhicheng Zhao, Fei Su, Bojin Zhuang, Nikolaos V. Boulgouris

Predicting where a pedestrian will step or how a car will merge is a fundamental challenge for autonomous driving. While humans perform these tasks instinctively, AI systems often struggle with the dynamic nature of real-world traffic, where objects frequently enter or exit the frame and the ego-vehicle itself is in motion.

A research team has proposed a novel framework that integrates object detection with "interaction reasoning" to predict the future positions and velocities of multiple targets simultaneously. By combining spatial context with temporal motion features, the system aims to navigate the complexities of urban environments more effectively than traditional sequence-based models.

Illustration of the prediction task, mapping observed RGB and optical-flow data to future object states.

The Challenge of Real-World Traffic

Most existing future state prediction models are designed for controlled environments with static backgrounds and a fixed number of objects. However, real-world driving presents two major hurdles:

  1. Ambiguity: A single static frame cannot distinguish between a parked car and one moving at speed. Temporal context is required to resolve these ambiguities.
  2. Dynamic Complexity: Street scenes are "open-world" scenarios where the number of objects changes constantly. Standard neural networks often require fixed input sizes, making them brittle when the number of detected entities fluctuates.

Proposed Framework: Two-Stream Adaptive Architecture

The researchers introduced a two-stream architecture that utilizes stage-wise learning to process spatial and temporal information in parallel.

Overview of the proposed two-stream architecture featuring spatial and temporal processing with a feedback loop.

Spatial and Temporal Streams

The Spatial Stream employs a Single Shot MultiBox Detector (SSD) to identify objects such as cars, buses, and pedestrians, providing their precise coordinates. Simultaneously, the Temporal Stream uses TVNet to generate optical-flow-like features. These features capture pixel-level movement between consecutive frames, allowing the system to understand both the ego-motion of the vehicle and the relative velocities of surrounding targets.

Feedback and Interaction

A key innovation is the Feedback Mechanism. If the spatial detector exhibits low confidence—perhaps due to motion blur or low resolution—a feedback loop adjusts the parameters of the motion generator to compensate.

The "distilled" features from both streams are then fed into a modified Visual Interaction Network (VIN). Unlike standard VINs, this version incorporates an adaptive-pooling layer and an interface class to handle a varying number of detected objects. This allows the model to simulate how different vehicles interact and influence each other's future trajectories, rather than predicting each path in isolation.

Experimental Results

The model was evaluated on the Udacity (CrowdAI and Autti) datasets, which contain thousands of frames of complex urban driving and occlusion scenarios. The results highlight the efficacy of the interaction-based approach:

  • Improved Accuracy: The proposed spatiotemporal model achieved a 79.45% mAP (Mean Average Precision) for state prediction, outperforming a spatial-only baseline (76.53%).
  • Superiority over LSTMs: The architecture surpassed a two-stream LSTM approach by 13.86%, suggesting that explicit interaction modeling is more effective for traffic prediction than simple sequence modeling.
  • Efficiency: By utilizing SSD for detection, the framework avoids the computational overhead of expensive region proposals, making it a viable candidate for real-time deployment on autonomous platforms.

Conclusion and Future Directions

By linking a detector's performance to a motion generator through a feedback loop, the system becomes self-correcting and resilient to varying video quality. While the model currently excels at short-term (half-second) horizons, future work will focus on extending predictions to several seconds and improving performance in scenes with extreme occlusions. Ultimately, these predictive capabilities are intended to drive wiser obstacle avoidance decisions in fully integrated autonomous agents.

Summary updated

Abstract

Predicting the state of dynamic objects in a real traffic environment is a key issue in autonomous driving vehicles. Various approaches have been proposed to learn the dynamics from visual observations with static background. However, minimal research has been conducted in a real traffic environment due to the complicated and changeable scenes. This paper proposes an adaptive multi-target future state prediction (position/velocity) method under autonomous driving conditions. In particular, an adaptive visual interaction method and control mechanism are introduced to overcome the change in the number of objects in continuous driving frames. In addition, a two-stream architecture with stage-wise learning is utilized for accurate object state prediction by simultaneously complementing spatial and temporal information. Experiments on two public challenging datasets, namely Udacity (CrowdAI) and Udacity (Autti), demonstrate the effectiveness of the proposed method on multi-target dynamic state prediction in a real traffic environment.
BibTeX
@article{du2019adaptive,
  author = {Li Du and Zixuan Wang and Leiquan Wang and Zhicheng Zhao and Fei Su and Bojin Zhuang and Nikolaos V. Boulgouris},
  title = {{Adaptive Visual Interaction Based Multi-Target Future State Prediction For Autonomous Driving Vehicles}},
  journal = {IEEE Trans. Vehicular Technology},
  year = {2019},
  doi = {10.1109/tvt.2019.2905598},
}
Figure