Learning Deep Representations Using Convolutional Auto-encoders with Symmetric Skip Connections

Jianfeng Dong, Xiao-Jiao Mao, Chunhua Shen, Yu-Bin Yang

TL;DR

This paper revitalizes unsupervised pre-training for modern deep learning by introducing Symmetric Skip Connections into convolutional auto-encoders. By forcing the network to reconstruct corrupted images while passing low-level details through shortcuts, the authors demonstrate that the model learns high-level abstract representations that significantly improve performance in image classification and semantic segmentation, especially when labeled data is scarce.

What Problem Does This Paper Address?

In the early days of deep learning, unsupervised pre-training (like using Restricted Boltzmann Machines) was essential to get deep networks to converge. Today, techniques like ReLU activations, better weight initialization, and massive labeled datasets (like ImageNet) allow us to train deep models from scratch using purely supervised learning.

However, labeled data remains expensive and time-consuming to produce, while unlabeled data is virtually infinite. The authors address a fundamental question: Can unsupervised pre-training still help modern deep architectures? Specifically, they tackle the "copying" problem in standard auto-encoders, where the network often learns to simply memorize pixel values rather than understanding the semantic structure of the image.

Methods

The authors propose a Symmetrically Connected Convolutional Auto-encoder. The architecture consists of two main parts:

  1. Encoder: A chain of convolutional layers that acts as a feature extractor, using stride-2 convolutions instead of pooling to preserve spatial information.
  2. Decoder: A symmetric chain of deconvolutional layers that reconstructs the "clean" image from a corrupted version (denoising).

The "secret sauce" is the Symmetric Shortcut Connections. These links connect corresponding encoder and decoder layers via element-wise addition.

The architecture features symmetric skip connections between encoder and decoder layers.

During pre-training, the network is fed images corrupted by either Gaussian noise or block masking (inpainting). It is trained end-to-end to minimize the mean square error between the original and reconstructed images. Once pre-trained, the encoder (and sometimes the decoder) is fine-tuned on a specific supervised task.

Datasets

The researchers evaluated their method across several standard benchmarks:

  • CIFAR-10 & CIFAR-100: Used for general image classification and to test "initialization as regularization" (pre-training on the same dataset used for fine-tuning).
  • STL-10: A dataset specifically designed for unsupervised/semi-supervised learning, featuring 100,000 unlabeled images.
  • ImageNet 2012: Used for large-scale unsupervised pre-training.
  • PASCAL VOC 2007 & 2012: Used for evaluating transfer learning in classification and semantic segmentation.

Key Insights and Ideas

  • Shortcuts as Semantic Enablers: Standard auto-encoders struggle to learn abstract features because the bottleneck must carry all information, including low-level details needed for reconstruction. By adding shortcuts, low-level details "bypass" the deep layers, forcing the deepest parts of the encoder to focus on learning abstract, semantic representations.
  • Initialization as Regularization: Even when no extra unlabeled data is available, pre-training on the training set itself acts as a powerful regularizer, leading to better generalization than random initialization.
  • Emergent Semantic Sensitivity: Visualizations show that without any labels, a network pre-trained on animal images naturally develops "dog face detectors," while one pre-trained on vehicles becomes sensitive to wheels.
Ablation studies show that with shortcuts (green), classification accuracy improves with depth, unlike the version without shortcuts (red).

Key Results

  • Semi-Supervised Learning: On CIFAR-10 with only 4,000 labels, the method achieved 80.22% accuracy, outperforming several complex jointly-trained semi-supervised frameworks.
  • STL-10 Benchmark: The model reached 75.8% accuracy, setting a competitive mark for unsupervised pre-training methods.
  • Semantic Segmentation: The authors found that pre-training the decoder as well as the encoder leads to significant gains in segmentation tasks. Initializing both parts via pre-training yielded the highest Mean IoU on PASCAL VOC 2012.
Pre-training both the encoder and decoder leads to the best results in semantic segmentation tasks.

Limitations and Future Work

  • ResNet Integration: The authors noted difficulty in combining their pre-training strategy with Residual Networks (ResNet). They observed that ResNet’s heavy reliance on SGD and weight decay conflicted with the ADAM optimizer and generative nature of the auto-encoder pre-training.
  • Optimization Sensitivity: While shortcuts help, the 30-layer denoising network remains challenging to optimize.
  • Future Directions: The authors suggest further research into how different optimizers (SGD vs. ADAM) and regularization techniques (like weight decay) interact with reconstruction-based pre-training to unlock even better performance for state-of-the-art architectures.

Summary updated

Abstract

Unsupervised pre-training was a critical technique for training deep neural networks years ago. With sufficient labeled data and modern training techniques, it is possible to train very deep neural networks from scratch in a purely supervised manner nowadays. However, unlabeled data is easier to obtain and usually of very large scale. How to make use of them better to help supervised learning is still a well-valued topic. In this paper, we investigate convolutional denoising auto-encoders to show that unsupervised pre-training can still improve the performance of high-level image related tasks such as image classification and semantic segmentation. The architecture we use is a convolutional auto-encoder network with symmetric shortcut connections. We empirically show that symmetric shortcut connections are very important for learning abstract representations via image reconstruction. When no extra unlabeled data are available, unsupervised pre-training with our network can regularize the supervised training and therefore lead to better generalization performance. With the help of unsupervised pre-training, our method achieves very competitive results in image classification using very simple all-convolution networks. When labeled data are limited but extra unlabeled data are available, our method achieves good results in several semi-supervised learning tasks.
BibTeX
@inproceedings{dong2016learning,
  author = {Jianfeng Dong and Xiao-Jiao Mao and Chunhua Shen and Yu-Bin Yang},
  title = {{Learning Deep Representations Using Convolutional Auto-encoders with Symmetric Skip Connections}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2016},
}
Figure