Training Group Orthogonal Neural Networks with Privileged Information

Yunpeng Chen, Xiaojie Jin, Jiashi Feng, Shuicheng Yan

Abstract

Learning rich and diverse representations is critical for the performance of deep convolutional neural networks (CNNs). In this paper, we consider how to use privileged information to promote inherent diversity of a single CNN model such that the model can learn better representations and offer stronger generalization ability. To this end, we propose a novel group orthogonal convolutional neural network (GoCNN) that learns untangled representations within each layer by exploiting provided privileged information and enhances representation diversity effectively. We take image classification as an example where image segmentation annotations are used as privileged information during the training process. Experiments on two benchmark datasets -- ImageNet and PASCAL VOC -- clearly demonstrate the strong generalization ability of our proposed GoCNN model. On the ImageNet dataset, GoCNN improves the performance of state-of-the-art ResNet-152 model by absolute value of 1.2% while only uses privileged information of 10% of the training images, confirming effectiveness of GoCNN on utilizing available privileged knowledge to train better CNNs.

Introduction

Deep learning models often suffer from feature redundancy, where different neurons learn similar patterns, limiting the model's overall representational power. While ensembling multiple models can mitigate this, it is computationally expensive. This research explores how to embed diversity into a single model using Learning Using Privileged Information (LUPI). Privileged information, such as pixel-level segmentation masks, is often available during training but absent during deployment. The authors propose a principled way to leverage this "extra" data to build a superior standard classifier.

Methodology: The GoCNN Architecture

The core innovation is the Group Orthogonal Convolutional Neural Network (GoCNN) architecture, which introduces a group orthogonal constraint to force the learning of non-redundant features.

  1. Feature Splitting: The final convolutional layer is divided into two groups: a foreground group and a background group (typically in a 3:1 ratio).
  2. Suppression Loss: During training, segmentation masks guide a "suppression" regression loss. Foreground filters are penalized for activating on background pixels, and vice versa. This forces the two groups to become "orthogonal," learning uncorrelated, distinct features.
  3. Multi-Task Supervision: The model is trained with three heads: a foreground classifier, a background classifier, and a "Main Classifier" that integrates both.
  4. Inference: At test time, the masks and auxiliary classifiers are discarded. The model functions as a standard CNN with zero extra computational overhead.
Architecture of GoCNN illustrating feature splitting and suppression loss using segmentation masks.

Experimental Evaluation

The researchers evaluated GoCNN on ImageNet and PASCAL VOC 2012. For ImageNet, they created "ImageNet-0.1m" with 130,000 manually annotated masks and tested on the full 1.2 million image set where only 10% of data had privileged masks.

Key Results

  • Performance Gains: On the ResNet-152 architecture, GoCNN improved Top-1 accuracy by 1.2% absolute on ImageNet-Full. For a ResNet-18 baseline on ImageNet-0.1m, accuracy jumped from 46.00% to 50.39%.
  • Generalization: Features learned on ImageNet showed superior transferability. When fine-tuned on PASCAL VOC, GoCNN outperformed vanilla ResNet-18 by 1.9% mAP (79.4% vs 77.5%).
  • Visual Analysis: Activation maps provide qualitative proof of the method's efficacy. GoCNN successfully separates objects from their surroundings, whereas standard ResNets often exhibit "bleeding" activations that conflate foreground and background.
Visual comparison of activation maps between standard CNNs and GoCNN, showing clearer separation of foreground and background.

Discussion and Insights

A significant finding is that explicitly learning background features—rather than treating background as noise—helps the model recognize foreground objects more effectively. By "cleaning" foreground features of background contamination, the representations become more robust.

Furthermore, the method is highly effective in semi-supervised settings. Even when only 10% of the training data includes masks, the privileged samples guide the network toward a better feature space that benefits unprivileged samples as well.

Limitations and Future Work

While effective, the approach requires high-quality segmentation masks for a portion of the training data, which is more labor-intensive than simple labeling. Additionally, the 3:1 foreground-to-background filter ratio was determined empirically; different tasks might benefit from different balances. Future research could explore applying group orthogonal constraints across multiple layers rather than just the final convolutional layer.

Summary updated

Abstract

Learning rich and diverse representations is critical for the performance of deep convolutional neural networks (CNNs). In this paper, we consider how to use privileged information to promote inherent diversity of a single CNN model such that the model can learn better representations and offer stronger generalization ability. To this end, we propose a novel group orthogonal convolutional neural network (GoCNN) that learns untangled representations within each layer by exploiting provided privileged information and enhances representation diversity effectively. We take image classification as an example where image segmentation annotations are used as privileged information during the training process. Experiments on two benchmark datasets -- ImageNet and PASCAL VOC -- clearly demonstrate the strong generalization ability of our proposed GoCNN model. On the ImageNet dataset, GoCNN improves the performance of state-of-the-art ResNet-152 model by absolute value of 1.2% while only uses privileged information of 10% of the training images, confirming effectiveness of GoCNN on utilizing available privileged knowledge to train better CNNs.
BibTeX
@inproceedings{chen2017training,
  author = {Yunpeng Chen and Xiaojie Jin and Jiashi Feng and Shuicheng Yan},
  title = {{Training Group Orthogonal Neural Networks with Privileged Information}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.24963/ijcai.2017/212},
}
Figure