Deep Trans-layer Unsupervised Networks for Representation Learning

Wentao Zhu, Jun Miao, Laiyun Qing, Xilin ChenarXiv 2015 · OpenReview

TL;DR

This paper introduces a novel "deep trans-layer unsupervised network" that learns powerful feature representations from unlabeled data without requiring extensive fine-tuning. By concatenating feature maps from lower layers to higher layers and incorporating local contrast normalization and whitening, the method achieves competitive performance on various recognition tasks with significantly fewer parameters than traditional deep learning approaches.

What Problem Does This Paper Address?

In machine learning, learning effective feature representations from vast amounts of unlabeled data is crucial for high-level tasks like classification and recognition. While deep learning models have achieved great success, they often require complex architectures, numerous parameters, and extensive fine-tuning, demanding significant expertise and computational resources. The authors observe that filters learned by these complex models often resemble simpler, hand-crafted features. They aim to bridge this gap by developing an unsupervised feature learning method that leverages simpler building blocks (like PCA or autoencoders) but enhances their representational power through a novel "trans-layer" connection strategy, reducing the need for fine-tuning and parameter tuning.

Methods

The proposed "Deep Trans-layer Unsupervised Network" (DTUN) is a three-layer architecture.

  • Unsupervised Learning Blocks: Each layer uses standard unsupervised learning methods like Principal Component Analysis (PCA) or autoencoders to learn local receptive filter banks.
  • Pre-processing: Before unsupervised learning in each layer, patches are pre-processed using Local Contrast Normalization (LCN) and whitening. LCN helps normalize for brightness and contrast, while whitening decorrelates features.
  • Trans-layer Connection: This is the core innovation. Instead of simply stacking layers, the feature maps from the first layer are concatenated with the feature maps from the second layer. This "trans-layer" approach aims to preserve more information from earlier stages, mitigating the information loss that can occur in traditional deep, sequential architectures.
  • Output Layer: The concatenated feature maps are then binarized and processed through a block-wise histogram to create translation and rotation invariant representations. This final representation is then used for classification.

Datasets

The paper evaluates its method on several standard datasets:

  • MNIST: For digit recognition.
  • MNIST Variations: A suite of datasets derived from MNIST, designed to test invariance to rotation, background noise, and image backgrounds.
  • Caltech 101: For object recognition.
  • LFW-a (Labeled Faces in the Wild): For face verification.

Key Insights and Ideas

  • Information Preservation is Key: The "trans-layer" concatenation is crucial. It directly addresses the problem of information loss in deep, sequential processing, allowing lower-level details to contribute to the final representation.
  • Simplicity Can Be Powerful: By using simpler unsupervised learning methods (PCA, autoencoders) as building blocks and enhancing them with clever architectural design (trans-layer) and pre-processing (LCN, whitening), the method achieves strong results without the complexity of traditional deep networks.
  • Pre-processing Matters: LCN and whitening are shown to significantly boost the performance of the unsupervised learning components, making them more robust to variations in illumination and contrast.
  • No Fine-Tuning Needed: A major advantage is the elimination of the need for backpropagation-based fine-tuning, which drastically reduces computational cost and expertise required.

Key Results

The DTUN demonstrates strong performance across various tasks:

  • MNIST: Achieved 99.45% accuracy.
  • Caltech 101: Reached 67.11% accuracy with 15 samples per class and 75.98% with 30 samples per class.
  • LFW-a: Achieved 87.10% verification accuracy.

The paper also shows that the trans-layer connection consistently improves performance over models without it, and LCN also provides benefits, especially on natural images. The proposed method uses significantly fewer parameters compared to traditional deep learning models.

Limitations and Future Work

  • High Dimensionality: The paper acknowledges that the trans-layer representation can still be high-dimensional, which can be a challenge for some applications.
  • Future Work: The authors suggest exploring other methods for preserving translation and rotation invariance to further reduce the dimensionality of the representations. They also hint at the potential for this trans-layer architecture to inspire new directions in unsupervised learning.

Summary updated

Abstract

Learning features from massive unlabelled data is a vast prevalent topic for high-level tasks in many machine learning applications. The recent great improvements on benchmark data sets achieved by increasingly complex unsupervised learning methods and deep learning models with lots of parameters usually requires many tedious tricks and much expertise to tune. However, filters learned by these complex architectures are quite similar to standard hand-crafted features visually. In this paper, unsupervised learning methods, such as PCA or auto-encoder, are employed as the building block to learn filter banks at each layer. The lower layer responses are transferred to the last layer (trans-layer) to form a more complete representation retaining more information. In addition, some beneficial methods such as local contrast normalization and whitening are added to the proposed deep trans-layer networks to further boost performance. The trans-layer representations are followed by block histograms with binary encoder schema to learn translation and rotation invariant representations, which are utilized to do high-level tasks such as recognition and classification. Compared to traditional deep learning methods, the implemented feature learning method has much less parameters and is validated in several typical experiments, such as digit recognition on MNIST and MNIST variations, object recognition on Caltech 101 dataset and face verification on LFW dataset. The deep trans-layer unsupervised learning achieves 99.45% accuracy on MNIST dataset, 67.11% accuracy on 15 samples per class and 75.98% accuracy on 30 samples per class on Caltech 101 dataset, 87.10% on LFW dataset.
BibTeX
@misc{zhu2015deep,
  author = {Wentao Zhu and Jun Miao and Laiyun Qing and Xilin Chen},
  title = {{Deep Trans-layer Unsupervised Networks for Representation Learning}},
  year = {2015},
  eprint = {1509.08038},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1509.08038},
}
Figure