Convolutional Memory Blocks for Depth Data Representation Learning

Keze Wang, Liang Lin, Chuangjie Ren, Wei Zhang, Wenxiu Sun

Introduction

Depth sensors, such as the Microsoft Kinect, provide valuable 3D information but present unique challenges for computer vision. Unlike RGB images, depth maps lack color and texture, making them "feature-poor." This scarcity of discriminative cues often forces standard Convolutional Neural Networks (CNNs) to rely on massive amounts of labeled data to distinguish complex structures like human body parts.

To address this, researchers introduced the Convolutional Memory Block (CMB). This module enables CNNs to learn more efficiently by capturing and "remembering" structural correlations across non-sequential samples in a dataset, effectively creating a global cache of representative features.

The Convolutional Memory Block (CMB)

The core innovation is a specialized module that maintains an internal memory of feature maps, updated during training. Unlike traditional Recurrent Neural Networks (RNNs) that handle sequential time-series data, the CMB is designed to exploit structural similarities across the entire depth dataset.

Architecture of the Convolutional Memory Block (CMB), illustrating the flow between input, memory, and output controllers across training iterations.

The CMB architecture consists of three primary components:

  1. Input Controller: Extracts internal representations from the current input feature map.
  2. Memory Controller: Acts as the central processing unit, deciding how to fuse existing memory with new input. It employs a "similarity-based fetching" mechanism to ensure only relevant memory items are integrated.
  3. Output Controller: Combines the updated memory with the original input to produce an enhanced feature representation.

By using hierarchical 3x3 and 1x1 convolutions, the CMB extracts high-level abstract information while maintaining a low parameter count, avoiding the computational overhead of fully connected memory layers.

Experimental Evaluation and Results

The framework was evaluated on several benchmarks, including the large-scale K2HPD (Kinect2 Human Pose Dataset), the ITOP (Invariant-Top View) dataset, and the small-scale ASTAR hand-depth dataset.

Data Efficiency and Background Suppression

A key finding was the model's remarkable data efficiency. A CMB-equipped network trained on just 10% of the data outperformed a standard CNN trained on the full dataset. Visualization of the internal memory maps revealed that the block naturally learns to suppress noisy background regions while focusing on the consistent structural patterns of the human body.

State-of-the-Art Performance

When integrated into a lightweight "hourglass" architecture, the model achieved superior accuracy and speed:

  • Accuracy: On the K2HPD benchmark, it reached an average Percent of Detected Joints (PDJ) of 84.9%, significantly outperforming Stacked Hourglass (72.1%) and Convolutional Pose Machines (67.5%).
  • Efficiency: The model operates at 70 FPS on a single GPU and has a compact footprint of only 58MB, making it suitable for real-time applications.
Qualitative comparison of human pose estimation results. The proposed method ("Ours") shows closer alignment to the ground truth than existing CPM and SH models.

Conclusion

The Convolutional Memory Block provides a powerful mechanism for enhancing depth data representation. By leveraging non-sequential structural correlations, it overcomes the inherent limitations of feature-poor depth maps. The resulting models are not only more accurate but also significantly more data-efficient and faster than traditional heavy-duty architectures. Future work may involve extending this memory mechanism to depth video data to combine temporal and structural dependencies.

Summary updated

Abstract

Compared to natural RGB images, data captured by 3D / depth sensors (e.g., Microsoft Kinect) have different properties, e.g., less discriminable in appearance due to lacking color / texture information. Applying convolutional neural networks (CNNs) on these depth data would lead to unsatisfying learning efficiency, i.e., requiring large amounts of annotated training data for convergence. To address this issue, this paper proposes a novel memory network module, called Convolutional Memory Block (CMB), which empowers CNNs with the memory mechanism on handling depth data. Different from the existing memory networks that store long / short term dependency from sequential data, our proposed CMB focuses on modeling the representative dependency (correlation) among non-sequential samples. Specifically, our CMB consists of one internal memory (i.e., a set of feature maps) and three specific controllers, which enable a powerful yet efficient memory manipulation mechanism. In this way, the internal memory, being implicitly aggregated from all previous inputted samples, can learn to store and utilize representative features among the samples. Furthermore, we employ our CMB to develop a concise framework for predicting articulated pose from still depth images. Comprehensive evaluations on three public benchmarks demonstrate significant superiority (about 6%) of our framework over all the compared methods. More importantly, thanks to the enhanced learning efficiency, our framework can still achieve satisfying results using 50% less training data.
BibTeX
@inproceedings{wang2018convolutional,
  author = {Keze Wang and Liang Lin and Chuangjie Ren and Wei Zhang and Wenxiu Sun},
  title = {{Convolutional Memory Blocks for Depth Data Representation Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.24963/ijcai.2018/387},
}
Figure