Scene Categorization with Spectral Features

Salman H. Khan, Munawar Hayat, Fatih Porikli

Abstract

Modern Convolutional Neural Networks (CNNs) excel at object recognition through spatial hierarchies but often struggle with fine-grained scene classification, where global context is paramount. This paper introduces a "Spectral Transformation Layer" that explicitly models spectral signatures—capturing the "gist" of a scene. By integrating frequency-domain information, the proposed architecture achieves significant performance gains in scene categorization and attribute detection while providing a built-in regularization effect through feature decorrelation.

Introduction

Scene recognition differs fundamentally from object recognition. While objects are defined by local parts and shapes, scenes are characterized by global layouts and "spatial envelope" properties like openness or naturalness. Early computer vision utilized handcrafted spectral features, such as the GIST descriptor, to capture these global signatures. However, the shift toward deep learning has prioritized spatial hierarchies, sometimes at the expense of global context. This work bridges the gap by reintroducing spectral analysis into CNN architectures to better distinguish between similar environments, such as bus versus train stations.

Methodology: The Spectral Transformation Layer

The core innovation is the Spectral Transformation Layer, typically inserted before the fully connected layers of a network. This layer applies a unitary transformation, specifically the Discrete Cosine Transform (DCT), to intermediate activations.

Architecture of the Spectral Transformation Layer showing the mapping from input feature maps to the spectral domain.

Technical Foundations

  • KLT Approximation: The authors demonstrate that for a large number of neurons, the Discrete Fourier Transform (DFT) asymptotically approximates the Karhunen-Loeve Transform (KLT). While KLT is the optimal method for data decorrelation, its complexity is prohibitive; spectral transforms provide a computationally efficient alternative.
  • Implementation: The transform is implemented as a $1 \times 1$ convolutional layer with fixed basis vectors, allowing seamless integration into standard architectures like VGGnet-16 and enabling end-to-end training via backpropagation.
  • Decorrelation and Regularization: By transforming activations into the frequency domain, the network is forced to represent information in independent components. This reduces "co-adaptation" between neurons, serving as a principled regularizer that improves model robustness.

Experimental Results

The approach was evaluated on several large-scale benchmarks, including MIT-67 (indoor scenes), SUN-397, and Places-205 (2.5 million images).

The addition of spectral features led to consistent accuracy improvements:

  • MIT-67: Increased from 80.9% to 84.3%.
  • Places-205: Rose from 60.3% to 61.2%.
  • SUN-397: Improved from 66.9% to 67.6%.
t-SNE visualization of feature distributions: the baseline (left) shows significant overlap, while the spectral features (right) exhibit clearer clustering.

In attribute detection tasks, the method achieved state-of-the-art results, reaching 90.3% mAP on the SUN Attribute dataset. Ablation studies confirmed that these gains were specifically due to the spectral transformation rather than simply increasing network depth.

Discussion and Key Insights

Analysis of the spectral features revealed several key findings:

  1. The "Gist" Effect: Spectral features naturally align with human-interpretable attributes (e.g., "man-made," "cluttered"), capturing the scene's essence at a "first glance."
  2. Magnitude Dominance: In Fourier analysis, magnitude information was found to be the primary driver for classification, while phase information often introduced noise.
  3. Domain Transfer: Spectral features proved highly effective in transfer learning scenarios, such as applying ImageNet-trained models to scene recognition tasks.

Conclusion and Future Work

By reintroducing frequency-domain analysis into deep learning, this work enhances the ability of CNNs to capture global scene context. Future research may explore learning optimal unitary transformations rather than using fixed bases and optimizing these layers for real-time applications. Testing the spectral layer within modern architectures like Vision Transformers (ViTs) remains a promising direction for further validation.

Summary updated

Abstract

Spectral signatures of natural scenes were earlier found to be distinctive for different scene types with varying spatial envelope properties such as openness, naturalness, ruggedness, and symmetry. Recently, such handcrafted features have been outclassed by deep learning based representations. This paper proposes a novel spectral description of convolution features, implemented efficiently as a unitary transformation within deep network architectures. To the best of our knowledge, this is the first attempt to use deep learning based spectral features explicitly for image classification task. We show that the spectral transformation decorrelates convolutional activations, which reduces co-adaptation between feature detections, thus acts as an effective regularizer. Our approach achieves significant improvements on three large-scale scene-centric datasets (MIT-67, SUN-397, and Places-205). Furthermore, we evaluated the proposed approach on the attribute detection task where its superior performance manifests its relevance to semantically meaningful characteristics of natural scenes.
BibTeX
@inproceedings{khan2017scene,
  author = {Salman H. Khan and Munawar Hayat and Fatih Porikli},
  title = {{Scene Categorization with Spectral Features}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2017},
  doi = {10.1109/iccv.2017.601},
}
Figure