Deep Anticipation: Light Weight Intelligent Mobile Sensing in IoT by Recurrent Architecture

Guang Chen, Shu Liu, Gereon Hinz, Zhongnan Qu, Alois C. KnollarXiv 2017 · OpenReview

TL;DR

This paper introduces the Kalman GRU Array (KGA), a novel lightweight recurrent neural network architecture designed for intelligent mobile sensing in the Internet of Things (IoT). KGA combines Gated Recurrent Units (GRUs) with Kalman filtering to predict the motion and interactions of objects in dynamic environments, even with noisy or incomplete sensor data, paving the way for more robust autonomous systems.

What Problem Does This Paper Address?

The proliferation of the Internet of Things (IoT) era promises a world where devices constantly communicate, enabling sophisticated mobile sensing for applications like autonomous vehicles and robotics. However, this rich data stream presents significant challenges: high latency, packet loss, and measurement noise can severely degrade the reliability of sensor information. These issues can lead to critical errors, such as missing a pedestrian in a complex traffic scenario. Existing methods often struggle with these real-world imperfections, especially when dealing with interactions between multiple objects. This paper aims to address these challenges by developing an intelligent system that can effectively interpret and anticipate dynamic scenes despite these data quality issues.

Methods

The core of the proposed solution is the Kalman GRU Array (KGA). This architecture integrates two key components:

  • Gated Recurrent Unit (GRU) Array: Each grid cell in an occupancy map is assigned a GRU cell, allowing the model to learn the temporal dynamics of that cell's occupancy. To capture spatial context, a "spatio-pooling" strategy is employed, where a cell's state is influenced by its neighbors. This is achieved using convolutional neural networks to pool hidden states.
  • Kalman Filtering Integration: The GRU array's predictions are combined with a Kalman-like prediction-correction mechanism. This allows the model to recursively update its belief about the scene's state, balancing predictions from the GRU with incoming (potentially noisy) measurements. The model learns to develop a "biased belief" between prediction and measurement, adapting to different levels of data reliability.

The authors also compare their approach to a Convolutional GRU (ConvGRU), a more parameter-heavy state-of-the-art architecture.

Datasets

The paper evaluates the KGA model on:

  • Synthetic Datasets: Generated using the Boids algorithm, simulating flocking behavior with an emphasis on avoidance functions.
  • Real-World Datasets:
    • UCY Dataset: Contains pedestrian movement data.
  • NGSIM Dataset: Includes vehicle trajectory data, specifically focusing on an intersection scene (Lankershim) to capture avoidance maneuvers.

The datasets are organized as videos, with each frame represented as a 50x50 binary occupancy grid. To simulate real-world challenges, "miss noise" (objects missing from input frames) and "shift noise" (objects' perceived locations deviating from ground truth) were introduced.

Key Insights and Ideas

  • Bridging the Gap between Prediction and Observation: The Kalman filtering integration is crucial. It allows the model to intelligently weigh its own predictions against incoming sensor data, making it robust to situations where one is more reliable than the other (e.g., when sensor data is highly noisy, the model relies more on its learned predictions).
  • Spatio-Temporal Learning for Dynamics: By combining GRUs for temporal modeling with spatio-pooling for spatial context, the model can learn complex environmental dynamics, including object interactions.
  • Lightweight Architecture for Mobile IoT: The KGA's design, particularly the GRU array and efficient parameter sharing, results in a significantly smaller model compared to ConvGRU, making it suitable for resource-constrained mobile devices.
  • Anticipating Interactions: The model demonstrates the ability to anticipate object interactions, such as predicting potential collisions and the resulting avoidance maneuvers.

Key Results

  • Comparable Performance: KGA achieves comparable performance to the more complex ConvGRU model on both synthetic and real-world datasets.
  • Significant Parameter Reduction: KGA uses only about 10% of the trainable parameters of ConvGRU (3906 vs. 30626).
  • Faster Inference: KGA is approximately 3 times faster than ConvGRU, with an inference time of about 5 ms per frame compared to 18 ms per frame on a CPU setup.
  • Robustness to Noise: The model effectively handles simulated miss noise (up to 80%) and shift noise (up to 10%), demonstrating its ability to infer true dynamics even with significant data degradation.

Limitations and Future Work

  • Inhomogeneous Scenes: While the current GRU array assumes scene homogeneity, the authors suggest that deploying different GRU cells for different parts of a scene could be a future improvement for inhomogeneous environments.
  • Unsupervised Training: The authors plan to explore unsupervised training methods for future work.
  • Broader Applications: They envision KGA's applicability extending to other spatio-temporally correlated tasks like human action recognition, video stabilization, and image generation.

Summary updated

Abstract

The rapid growth of IoT era is shaping the future of mobile services. Advanced communication technology enables a heterogeneous connectivity where mobile devices broadcast information to everything. Mobile applications such as robotics and vehicles connecting to cloud and surroundings transfer the short-range on-board sensor perception system to long-range mobile-sensing perception system. However, the mobile sensing perception brings new challenges for how to efficiently analyze and intelligently interpret the deluge of IoT data in mission- critical services. In this article, we model the challenges as latency, packet loss and measurement noise which severely deteriorate the reliability and quality of IoT data. We integrate the artificial intelligence into IoT to tackle these challenges. We propose a novel architecture that leverages recurrent neural networks (RNN) and Kalman filtering to anticipate motions and interac- tions between objects. The basic idea is to learn environment dynamics by recurrent networks. To improve the robustness of IoT communication, we use the idea of Kalman filtering and deploy a prediction and correction step. In this way, the architecture learns to develop a biased belief between prediction and measurement in the different situation. We demonstrate our approach with synthetic and real-world datasets with noise that mimics the challenges of IoT communications. Our method brings a new level of IoT intelligence. It is also lightweight compared to other state-of-the-art convolutional recurrent architecture and is ideally suitable for the resource-limited mobile applications.
BibTeX
@misc{chen2017deep,
  author = {Guang Chen and Shu Liu and Gereon Hinz and Zhongnan Qu and Alois C. Knoll},
  title = {{Deep Anticipation: Light Weight Intelligent Mobile Sensing in IoT by Recurrent Architecture}},
  year = {2017},
  eprint = {1801.01444},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1801.01444},
}
Figure