Learning Deep Representations Using Convolutional Auto-encoders with Symmetric Skip Connections

Jianfeng Dong, Xiao-Jiao Mao, Chunhua Shen, Yu-Bin Yang

Abstract

Unsupervised pre-training, once a cornerstone of deep learning, has largely been supplanted by supervised training on massive labeled datasets. However, the abundance of unlabeled data remains an untapped resource. This paper investigates convolutional denoising auto-encoders (CDAEs) with symmetric shortcut connections, demonstrating that unsupervised pre-training can significantly enhance performance in high-level tasks like image classification and semantic segmentation. We show that these skip connections are vital for learning abstract representations and that pre-training serves as an effective regularizer, especially when labeled data is limited.

Introduction: The Role of Unsupervised Pre-training

In the early era of deep learning, unsupervised pre-training was essential for network convergence. Modern advancements like Batch Normalization and specialized weight initialization have made supervised training from scratch the standard. Yet, labeled data is expensive. The authors revisit unsupervised pre-training to ask if it can still benefit modern deep convolutional architectures. The goal is to move beyond simple pixel-level reconstruction toward learning semantic, high-level features useful for downstream tasks.

Methodology: Symmetric Skip Connections

The core of the proposed approach is a Convolutional Denoising Auto-encoder (CDAE) featuring a symmetric architecture.

Network architecture and shortcut connection details.

The network consists of an encoder (convolutional layers) and a symmetric decoder (deconvolutional layers). The "secret sauce" is the Symmetric Skip Connection: every few layers, an encoder layer is linked to its corresponding decoder layer via element-wise addition. This design eases the optimization of the reconstruction task, allowing the network to focus on learning abstract features rather than just copying pixels.

The model is trained end-to-end using the ADAM optimizer to reconstruct clean images from corrupted inputs (e.g., Gaussian noise or pixel masking). Once pre-trained, the encoder (and sometimes the decoder) is used to initialize weights for supervised tasks.

Experiments and Results

The method was evaluated on benchmarks including CIFAR-10/100, STL-10, PASCAL VOC, and ImageNet.

Visual examples of image corruption and reconstruction.

Key Findings

  • Semi-Supervised Performance: On the STL-10 dataset, which features a large unlabeled set and a small labeled set, the method achieved 75.8% accuracy, surpassing many complex contemporary frameworks.
  • Regularization and Generalization: Even without extra unlabeled data, pre-training on the existing training set acts as a regularizer, providing a better weight initialization that improves generalization.
  • Transferring the Decoder: For semantic segmentation on PASCAL VOC 2012, transferring both the pre-trained encoder and decoder yielded significant gains in Mean Intersection over Union (mIoU) compared to standard encoder-only transfer.
  • Emergent Semantics: Visualizations revealed that the network naturally learns semantic concepts. For instance, models pre-trained on animal images developed "dog-face" detectors, while those trained on vehicles became sensitive to wheels, all without explicit labels.

Conclusion and Future Work

This research proves that unsupervised pre-training remains a powerful tool for modern deep learning, particularly for improving robustness and performance in data-scarce scenarios. While the authors noted challenges in combining this specific pre-training regime with standard ResNet architectures due to optimization conflicts (SGD vs. ADAM), the results highlight the enduring value of learning from unlabeled data through symmetric auto-encoders. Future work will focus on harmonizing these architectural and optimization strategies.

Summary updated

Abstract

Unsupervised pre-training was a critical technique for training deep neural networks years ago. With sufficient labeled data and modern training techniques, it is possible to train very deep neural networks from scratch in a purely supervised manner nowadays. However, unlabeled data is easier to obtain and usually of very large scale. How to make use of them better to help supervised learning is still a well-valued topic. In this paper, we investigate convolutional denoising auto-encoders to show that unsupervised pre-training can still improve the performance of high-level image related tasks such as image classification and semantic segmentation. The architecture we use is a convolutional auto-encoder network with symmetric shortcut connections. We empirically show that symmetric shortcut connections are very important for learning abstract representations via image reconstruction. When no extra unlabeled data are available, unsupervised pre-training with our network can regularize the supervised training and therefore lead to better generalization performance. With the help of unsupervised pre-training, our method achieves very competitive results in image classification using very simple all-convolution networks. When labeled data are limited but extra unlabeled data are available, our method achieves good results in several semi-supervised learning tasks.
BibTeX
@inproceedings{dong2016learning,
  author = {Jianfeng Dong and Xiao-Jiao Mao and Chunhua Shen and Yu-Bin Yang},
  title = {{Learning Deep Representations Using Convolutional Auto-encoders with Symmetric Skip Connections}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2016},
}
Figure