Super Interaction Neural Network

Yang Yao, Xu Zhang, Baile Xu, Furao Shen, Jian ZhaoarXiv 2019 · OpenReview

TL;DR

This paper introduces Super Interaction Neural Networks (SINet), a novel lightweight CNN architecture designed to boost performance by enhancing information flow. SINet achieves this through "Exchange Shortcut Connections" for cross-group feature interaction and "Dense Funnel Layers" and "Attention-based Hierarchical Joint Decision" for improved depth-wise feature utilization, outperforming existing lightweight models on ImageNet.

What Problem Does This Paper Address?

Modern deep learning models, particularly Convolutional Neural Networks (CNNs), have achieved remarkable success in computer vision. However, their high accuracy often comes at the cost of enormous computational resources, making them unsuitable for resource-constrained environments like mobile devices and IoT. Lightweight CNNs aim to address this by reducing model size and computation, but this often leads to shallower and thinner networks with limited feature information. The challenge is to design lightweight networks that can achieve high accuracy by maximizing the utilization of the available feature information, rather than simply relying on more complex architectures.

Methods

The authors propose three key innovations to enhance information interaction within lightweight CNNs:

  • Exchange Shortcut Connection: This method addresses the information bottleneck created by group convolutions. Instead of processing feature map groups independently, this technique allows information to be exchanged between groups without any additional computational cost. It's integrated into "SI Units," which are the building blocks of SINet.
  • Dense Funnel Layer: Inspired by DenseNet's dense connections, this layer aims to improve information flow along the depth of the network. It concatenates features from previous layers with the current layer's output and uses a 1x1 convolution to reduce dimensionality, thus avoiding the computational and memory explosion seen in standard DenseNets.
  • Attention-based Hierarchical Joint Decision: This approach leverages features from all middle layers for classification, rather than just the final layer. It uses an attention mechanism to assign adaptive weights to feature vectors derived from different network blocks, allowing the model to learn which hierarchical features are most important for the final decision.

These components are combined into "SI Units," which are then stacked into "SI Blocks" to form the overall SINet architecture.

Datasets

  • ImageNet: Used for training and evaluating the performance of the proposed SINet model against other state-of-the-art networks.
  • CIFAR-100: Used for ablation studies to verify the effectiveness and universality of the proposed components (SI Unit and Attention-based Hierarchical Joint Decision) when applied to different network architectures.

Key Insights and Ideas

  • Information Interaction is Key for Lightweight Networks: The paper strongly argues that for lightweight networks, maximizing the interaction and utilization of existing features is more critical than simply increasing depth or width.
  • Zero-Cost Feature Exchange: The "Exchange Shortcut Connection" is particularly noteworthy for achieving inter-group feature interaction without any computational overhead, a significant advantage for efficiency-focused models.
  • Balancing Dense Connections and Efficiency: The "Dense Funnel Layer" offers a clever way to achieve the benefits of dense connections (improved information flow) without the prohibitive computational cost, making it suitable for lightweight designs.
  • Hierarchical Feature Importance: The "Attention-based Hierarchical Joint Decision" highlights that different levels of features have varying importance for classification, and learning to dynamically weigh them can lead to better performance.

Key Results

On the ImageNet dataset, SINet demonstrates superior performance compared to other state-of-the-art lightweight models. For instance, SINet(1.8) achieves 74.8% Top-1 accuracy with 7.6M parameters and 570M FLOPs, outperforming MobileNetV2(1.4) and ShuffleNet(×2) while being competitive with NASNet-A. Ablation studies on CIFAR-100 confirm that the proposed SI Unit and Attention-based Hierarchical Joint Decision consistently improve accuracy across various network architectures, even general CNNs like ResNet-34, with minimal increases in parameters or computation.

Limitations and Future Work

The paper focuses on improving information interaction within lightweight CNNs. While SINet shows strong performance, the authors do not explicitly detail limitations or extensive future work beyond demonstrating the effectiveness of their components. However, the architecture's adaptability through a width multiplier suggests potential for further fine-tuning and exploration in different application domains. The universality of the proposed components across various architectures, as shown in ablation studies, hints at their broad applicability.

Summary updated

Abstract

Recent studies have demonstrated that the convolutional networks heavily rely on the quality and quantity of generated features. However, in lightweight networks, there are limited available feature information because these networks tend to be shallower and thinner due to the efficiency consideration. For farther improving the performance and accuracy of lightweight networks, we develop Super Interaction Neural Networks (SINet) model from a novel point of view: enhancing the information interaction in neural networks. In order to achieve information interaction along the width of the deep network, we propose Exchange Shortcut Connection, which can integrate the information from different convolution groups without any extra computation cost. And then, in order to achieve information interaction along the depth of the network, we proposed Dense Funnel Layer and Attention based Hierarchical Joint Decision, which are able to make full use of middle layer features. Our experiments show that the superior performance of SINet over other state-of-the-art lightweight models in ImageNet dataset. Furthermore, we also exhibit the effectiveness and universality of our proposed components by ablation studies.
BibTeX
@misc{yao2019super,
  author = {Yang Yao and Xu Zhang and Baile Xu and Furao Shen and Jian Zhao},
  title = {{Super Interaction Neural Network}},
  year = {2019},
  eprint = {1905.12349},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1905.12349},
}
Figure