Abstract
In this paper, we propose a new learning architecture named cosine activation in a compact network (CACN). The CACN is derived from kernel approximation and establishes a nonlinear hidden layer with the cosine activation function. By inheriting fusion ability in kernel approximation while learning parameters in a supervised way, the CACN is a well-directed solution to scene classification. By seamlessly connecting with convolutional neural networks (CNNs), it is easy to construct an end-to-end network. To compensate for the loss of spatial layout information in CNNs, the CACN is further combined with spatial pyramid matching to fuse various information into one holistic picture. The experiments on the MIT indoor and SUN397 datasets show that the CACN delivers high performance and demonstrates its great effectiveness for scene-classification tasks.
Introduction
While Convolutional Neural Networks (CNNs) have revolutionized object recognition, scene classification remains a distinct challenge. Unlike objects, which have well-defined boundaries, scenes (such as a "kitchen" or a "dining room") are defined by the global arrangement of multiple objects and textures.
Standard CNN architectures often struggle with scene classification for two primary reasons. First, the successive pooling layers in CNNs tend to discard precise spatial layout information, which is critical for understanding a scene's structure. Second, traditional architectures are not always optimized to "fuse" information from different parts or scales of an image into a single, discriminative representation. The Cosine Activation in Compact Network (CACN) addresses these issues by bridging the gap between classical kernel methods and modern deep learning.
The CACN Architecture
The core innovation of this research is the CACN layer, which features a hidden layer utilizing a cosine activation function. This design is mathematically grounded in "Random Fourier Features," a technique used to approximate complex kernels, such as the Radial Basis Function (RBF) kernel, in a lower-dimensional space.
Unlike traditional kernel approximation where parameters are often random and fixed, CACN learns these parameters (weights $W$ and biases $b$) via supervised learning. This ensures the "kernel" is specifically tuned to distinguish between scene categories. By replacing standard activation functions with this trainable cosine layer, the network can better capture complex spatial relationships and fuse diverse image features more effectively than standard CNNs alone.
Spatial Context and Feature Fusion
To address the loss of spatial information inherent in CNNs, the authors integrated Spatial Pyramid Matching (SPM) with the CACN. The process involves dividing an image into multiple grids (1x1, 2x2, and 4x4) and passing each through a CNN backbone (specifically VGG-16). The CACN layer then acts as the "glue," intelligently fusing these multi-scale spatial descriptors into a holistic scene summary.
Furthermore, the model employs a sophisticated objective function to ensure robustness. This includes:
- Sparsity Constraints ($l_{2,1}$ norm): Used to select the most relevant features for classification.
- Low-Rank Constraints (Nuclear Norm): Used to ensure the learned representations are robust and resistant to overfitting.
Experimental Results
The CACN approach was evaluated on two gold-standard benchmarks for scene recognition: the MIT Indoor dataset (67 categories) and the SUN397 dataset (397 categories).
- MIT Indoor: The full model (CNN + CACN + SPM) achieved an accuracy of 76.53%, representing a significant improvement over the 59.50% achieved by a standard VGG-16 baseline.
- SUN397: The model reached 56.78% accuracy. While the raw percentage is lower due to the 397-class complexity, it is highly competitive and outperformed several complex methods based on Fisher vectors and deep features.
- Efficiency: Testing revealed that a hidden layer size ($D$) of 1000 provided the optimal balance between classification accuracy and computational compactness.
Conclusion
The CACN architecture demonstrates that a simple cosine activation function can serve as a powerful feature-mapping tool, allowing networks to operate in a high-dimensional "kernel space" without prohibitive computational costs. While the current implementation uses VGG-16, the CACN layer could potentially be integrated with more modern backbones like ResNets or Vision Transformers. Future research may also focus on simplifying the optimization process, which currently requires an alternating SGD strategy to manage sparsity and low-rank constraints, or applying the architecture to video recognition and fine-grained retrieval tasks.
Summary updated