Introduction
Depth sensors, such as the Microsoft Kinect, provide valuable 3D information but present unique challenges for computer vision. Unlike RGB images, depth maps lack color and texture, making them "feature-poor." This scarcity of discriminative cues often forces standard Convolutional Neural Networks (CNNs) to rely on massive amounts of labeled data to distinguish complex structures like human body parts.
To address this, researchers introduced the Convolutional Memory Block (CMB). This module enables CNNs to learn more efficiently by capturing and "remembering" structural correlations across non-sequential samples in a dataset, effectively creating a global cache of representative features.
The Convolutional Memory Block (CMB)
The core innovation is a specialized module that maintains an internal memory of feature maps, updated during training. Unlike traditional Recurrent Neural Networks (RNNs) that handle sequential time-series data, the CMB is designed to exploit structural similarities across the entire depth dataset.
The CMB architecture consists of three primary components:
- Input Controller: Extracts internal representations from the current input feature map.
- Memory Controller: Acts as the central processing unit, deciding how to fuse existing memory with new input. It employs a "similarity-based fetching" mechanism to ensure only relevant memory items are integrated.
- Output Controller: Combines the updated memory with the original input to produce an enhanced feature representation.
By using hierarchical 3x3 and 1x1 convolutions, the CMB extracts high-level abstract information while maintaining a low parameter count, avoiding the computational overhead of fully connected memory layers.
Experimental Evaluation and Results
The framework was evaluated on several benchmarks, including the large-scale K2HPD (Kinect2 Human Pose Dataset), the ITOP (Invariant-Top View) dataset, and the small-scale ASTAR hand-depth dataset.
Data Efficiency and Background Suppression
A key finding was the model's remarkable data efficiency. A CMB-equipped network trained on just 10% of the data outperformed a standard CNN trained on the full dataset. Visualization of the internal memory maps revealed that the block naturally learns to suppress noisy background regions while focusing on the consistent structural patterns of the human body.
State-of-the-Art Performance
When integrated into a lightweight "hourglass" architecture, the model achieved superior accuracy and speed:
- Accuracy: On the K2HPD benchmark, it reached an average Percent of Detected Joints (PDJ) of 84.9%, significantly outperforming Stacked Hourglass (72.1%) and Convolutional Pose Machines (67.5%).
- Efficiency: The model operates at 70 FPS on a single GPU and has a compact footprint of only 58MB, making it suitable for real-time applications.
Conclusion
The Convolutional Memory Block provides a powerful mechanism for enhancing depth data representation. By leveraging non-sequential structural correlations, it overcomes the inherent limitations of feature-poor depth maps. The resulting models are not only more accurate but also significantly more data-efficient and faster than traditional heavy-duty architectures. Future work may involve extending this memory mechanism to depth video data to combine temporal and structural dependencies.
Summary updated