Efficient 3D Video Engine Using Frame Redundancy

Gao Peng, Bo Pang, Cewu LuWACV 2021

TL;DR

This paper introduces the Efficient 3D Video Engine (EVE), a novel framework that significantly reduces computational cost in 3D video understanding by exploiting the inherent redundancy between adjacent frames. EVE achieves this by intelligently reusing features from previously processed frames for geometrically similar points, leading to substantial speedups without sacrificing accuracy.

What Problem Does This Paper Address?

Understanding 3D videos, such as those generated by LiDAR sensors in autonomous driving or robotics, is computationally intensive. Current methods typically process each 3D point cloud frame independently, leading to redundant computations because adjacent frames in a video clip often share a high degree of geometric similarity. This redundancy makes real-time processing challenging and requires significant computational resources. The paper aims to address this by developing an efficient engine that leverages this frame-to-frame similarity to avoid unnecessary computations.

Methods

The core of EVE is a two-module approach:

  1. Redundancy Removing Module: This module identifies geometrically similar points between consecutive frames (Xt and Xt-1). It uses a matching function, specifically the Iterative Closest Point (ICP) algorithm, to find corresponding points. Points with a distance below a threshold are considered "matched" (P), while others are "unmatched" (Q). The ICP algorithm is preferred over simpler Nearest Neighbor (NN) matching for its robustness to camera motion and sampling variations.
  2. Residual Learning Module: For the matched points (P), EVE reuses the features computed for their corresponding points in the previous frame, incurring minimal computational cost. For the unmatched points (Q), a residual learning module is employed. Instead of processing these points from scratch, it learns the residual information by taking the output of the previous frame's features and adding a learned residual. This is formulated as $Y_t^- = f(X_t^Q) + f(X_{t-1}^Q)$, where $f$ represents a sparse convolutional network backbone. This residual approach helps preserve context while efficiently learning new information.

EVE is designed as a plug-and-play framework that can be easily integrated with existing 3D deep learning backbones, such as MinkUNet.

Datasets

The paper evaluates EVE on two datasets:

  • SemanticKITTI: A large-scale dataset for semantic scene understanding of LiDAR sequences, commonly used for 3D semantic segmentation tasks.
  • Synthia4D: A large synthetic dataset designed for visual inference in driving scenarios, providing dense and precise annotations for semantic classes and depth.

Key Insights and Ideas

  • Significant Frame Redundancy: The paper quantifies frame redundancy in 3D point cloud videos, showing that up to 86% of points in adjacent frames can have the same semantic label, indicating a massive opportunity for computational savings.
  • Geometry Implies Semantics: The core hypothesis is that geometrically similar points in adjacent frames likely share similar high-level semantic features, making direct recomputation unnecessary.
  • Residual Learning for Efficiency: Learning the difference or residual information for unmatched points, rather than their full features, is a more efficient way to capture new details without losing the context from the previous frame.
  • ICP for Robust Matching: The Iterative Closest Point (ICP) algorithm proves more effective than simpler methods like Nearest Neighbor for matching points between frames, leading to better performance.
  • Matching Ratio Trade-off: There's a sweet spot for the matching ratio (percentage of points considered redundant). Too low a ratio means less computation saved, while too high a ratio (e.g., >60%) can hinder performance as the residual learning module might not receive enough information.

Key Results

  • Computational Savings: EVE significantly reduces computation. For example, when compared to MinkUNet on SemanticKITTI, EVE achieves similar accuracy with up to 32.9 GFLOPs less computation.
  • Accuracy Maintenance/Improvement: EVE maintains or even improves segmentation accuracy. On SemanticKITTI, it achieves a 4.1 mIoU improvement over a strong baseline with similar computation. On Synthia4D, it shows a better performance/computation balance.
  • Efficiency: EVE is much faster than point-based methods. For instance, it reduces the average frame processing time from 2.01s (RandLA-Net) to 0.22s.
  • Category-wise Performance: EVE generally maintains performance across most semantic categories, with minor drops in some categories (e.g., "parking," "truck") and improvements in others (e.g., "bicyclist," "person").

Limitations and Future Work

  • Matching Accuracy: The paper highlights that matching accuracy is crucial. Errors in matching, especially for points near the road or with irregular patterns, can lead to failure cases.
  • Failure Cases: Qualitative results show that failure cases often occur between "road" and other categories like "sidewalk," suggesting that improving matching accuracy in these challenging scenarios is a future research direction.
  • Optimal Matching Ratio: While 50-60% matching ratio is found to be effective, further exploration into fine-tuning this parameter for different scenarios could be beneficial.
  • Temporal Depth: The paper explores using a limited number of frames (T). Increasing T generally leads to more computation saving, but the benefits plateau. Future work could explore more sophisticated ways to leverage longer temporal contexts.

Summary updated

BibTeX
@inproceedings{peng2021efficient,
  author = {Gao Peng and Bo Pang and Cewu Lu},
  title = {{Efficient 3D Video Engine Using Frame Redundancy}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
}
Figure