Learning Robust Representations with Information Bottleneck and Memory Network for RGB-D-based Gesture Recognition

Yunan Li, Huizhou Chen, Guanwen Feng, Qiguang MiaoICCV 2023 · OpenReview

Abstract

Although previous RGB-D-based gesture recognition methods have shown promising performance, researchers often overlook the interference of task-irrelevant cues like illumination and background. These unnecessary factors are learned together with the predictive ones by the network and hinder accurate recognition. In this paper, we propose a convenient and analytical framework to learn a robust feature representation that is impervious to gesture-irrelevant factors. Based on the Information Bottleneck theory, two rules of Sufficiency and Compactness are derived to develop a new information-theoretic loss function, which cultivates a more sufficient and compact representation from the feature encoding and mitigates the impact of gesture-irrelevant information. To highlight the predictive information, we further integrate a memory network. Using our proposed content-based and contextual memory addressing scheme, we weaken the nuisances while preserving the task-relevant information, providing guidance for refining the feature representation. Experiments conducted on three public datasets demonstrate that our approach leads to a better feature representation and achieves better performance than state-of-the-art methods.

Introduction

In gesture recognition, deep learning models frequently struggle with "nuisances"—factors unrelated to the gesture itself, such as the performer's attire, ambient lighting, or background clutter. Standard networks often memorize these irrelevant cues along with the actual hand movements, leading to high intra-class variance. For instance, the same "wave" gesture might appear vastly different to a model if filmed in different lighting conditions. The goal of this research is to disentangle predictive motion information from superfluous environmental data.

Methodology

The proposed framework, InBoMem, utilizes a two-pronged approach to cultivate robust representations by stripping away task-irrelevant noise.

Conceptual overview of the InBoMem framework, illustrating the separation of essential gesture features from task-irrelevant nuisances like illumination and background.

Information Bottleneck (IB) Objective

Based on Information Bottleneck theory, the authors derive a loss function governed by two principles:

  • Sufficiency: The learned features must retain enough information to correctly identify the gesture label.
  • Compactness: The features should be as compressed as possible, discarding any information that does not assist in prediction. By minimizing this loss, the network is forced to "squeeze" the input through a bottleneck, ideally retaining only essential gesture data.

Memory Network for Feature Refinement

To provide a "gold standard" for gesture appearance, a memory network is integrated into the architecture to filter the encoded features.

The InBoMem architecture, detailing the encoder, the memory network, and the specific processes for memory writing and reading.

As the model processes samples, it writes features into a memory bank. This process effectively "overlays" features of the same gesture class, averaging out random background noise while amplifying consistent motion patterns. A sophisticated two-level addressing scheme—Content-based and Contextual—is then used to read a refined, noise-free version of the gesture feature from the memory bank.

Experimental Evaluation

The model was evaluated on three major public RGB-D gesture datasets: Chalearn IsoGD, EgoGesture, and THU-READ. Despite using a relatively simple Res3D-18 backbone, InBoMem outperformed previous state-of-the-art (SOTA) methods:

  • IsoGD: Achieved 74.08% accuracy (RGB+D), significantly outperforming previous SOTA on RGB data specifically.
  • EgoGesture: Reached 95.72% accuracy, setting a new benchmark for egocentric recognition.
  • THU-READ: Achieved 90.51% accuracy, a substantial improvement over Transformer-based and I3D-based methods.

Ablation studies highlighted that while the IB objective improves robustness, the Memory Network provided the most substantial performance boost, demonstrating the effectiveness of the "overlay" filtering effect.

Conclusion

InBoMem demonstrates that the quality of feature representation is paramount in gesture recognition. By mathematically enforcing compactness and using a memory network to filter out environmental noise, the framework achieves superior performance with a standard backbone. Future research will address modality fusion and the application of this bottleneck approach to continuous gesture streams.

The code of our method is available at: https://github.com/Carpumpkin/InBoMem.

Summary updated

Abstract

Although previous RGB-D-based gesture recognition methods have shown promising performance, researchers often overlook the interference of task-irrelevant cues like illumination and background. These unnecessary factors are learned together with the predictive ones by the network and hinder accurate recognition. In this paper, we propose a convenient and analytical framework to learn a robust feature representation that is impervious to gesture-irrelevant factors. Based on the Information Bottleneck theory, two rules of Sufficiency and Compactness are derived to develop a new information-theoretic loss function, which cultivates a more sufficient and compact representation from the feature encoding and mitigates the impact of gesture-irrelevant information. To highlight the predictive information, we further integrate a memory network. Using our proposed content-based and contextual memory addressing scheme, we weaken the nuisances while preserving the task-relevant information, providing guidance for refining the feature representation. Experiments conducted on three public datasets demonstrate that our approach leads to a better feature representation and achieves better performance than state-of-the-art methods. The code of our method is available at: https://github.com/Carpumpkin/InBoMem.
BibTeX
@inproceedings{li2023learning,
  author = {Yunan Li and Huizhou Chen and Guanwen Feng and Qiguang Miao},
  title = {{Learning Robust Representations with Information Bottleneck and Memory Network for RGB-D-based Gesture Recognition}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=AGORPWQr2w},
}
Figure