TL;DR
This paper introduces the Wide Hidden Expansion (WHE) layer, a novel architectural component for deep convolutional neural networks that significantly increases the number of activation functions without a proportional increase in computational cost or memory usage. By strategically expanding the network's "implicit width" through a structured sparse connectivity, WHE layers consistently improve accuracy across various architectures and datasets, demonstrating a powerful new way to enhance non-linearity in deep learning models.
What Problem Does This Paper Address?
Deep convolutional neural networks (CNNs) rely on a combination of linear transformations (convolutional layers) and non-linear transformations (activation functions) to learn complex patterns in data. While increasing the depth of networks has been a primary focus for improving performance, this paper argues that increasing non-linearity is equally crucial. Existing methods for boosting non-linearity often involve designing more complex activation functions or increasing network depth, which can lead to higher computational and memory demands. The authors aim to address this by proposing a method to dramatically increase the number of activation functions within a CNN architecture while keeping computational and memory overheads minimal.
Methods
The core contribution is the Wide Hidden Expansion (WHE) layer. This layer is designed to be embedded within existing CNN architectures, typically after a convolutional layer. The key idea is to create a "wide hidden layer" where each channel is sparsely connected to a small number of input and output channels. Specifically, for an input group of g channels, the WHE layer generates g^2 hidden channels. Each of these hidden channels applies an activation function to a weighted sum of two input channels. These activated hidden channels are then aggregated to form the output channels.
This structured sparse connectivity is crucial:
- Increased Activation Functions: The number of activation functions grows quadratically with the group size
g, allowing for a massive increase in non-linearity. - Controlled Complexity: Despite the large number of hidden channels, the computational complexity is kept low because each hidden channel only interacts with a few input channels.
- Memory Efficiency: The paper highlights an implementation technique where the hidden layer is computed "on-the-fly" and cached in GPU shared memory, meaning it doesn't consume significant main GPU memory, aligning memory consumption with the input tensor size rather than the expanded hidden layer size.
Datasets
The paper evaluates the proposed WHE layer on the following datasets:
- ImageNet: A large-scale dataset for image classification (1000 categories).
- CIFAR-100: A smaller-scale image classification dataset (100 categories).
- Tiny-ImageNet: A subset of ImageNet, useful for faster prototyping.
- COCO: A dataset for object detection.
- Pascal VOC: Another dataset for object detection.
Key Insights and Ideas
- Implicit Width is Powerful: The WHE layer demonstrates that increasing the "implicit width" of a network (i.e., the number of intermediate non-linearities) can be as effective as increasing explicit depth or width, but with much lower overhead.
- Structured Sparsity is Key: The specific sparse connectivity pattern (each hidden channel connected to only two input channels) is what enables the massive increase in activation functions without a proportional computational cost. This is a key differentiator from methods like ResNeXt, which use grouped convolutions.
- On-the-Fly Computation for Memory Savings: The implementation detail of computing hidden layers on-the-fly is critical for making the WHE layer practical, preventing it from becoming a memory bottleneck.
- Universality: The WHE layer is shown to be a versatile building block, consistently improving performance when integrated into various established architectures like ResNet, SENet, and MobileNet, and even benefiting object detection models.
Key Results
- ImageNet Performance: Models with WHE layers achieved up to 2.01% higher Top-1 accuracy on ImageNet compared to baseline models, with less than 4% computation increase and less than 2% more parameters.
- CIFAR-100 and Tiny-ImageNet: Consistent improvements in accuracy were observed when applying WHE to ResNet models on these datasets.
- Object Detection: Embedding WHE into the ResNet backbone of the CenterNet object detection model boosted performance on COCO and Pascal VOC datasets.
- Comparison to Depthwise Convolutions: Replacing WHE layers with low-cost Depthwise convolutions yielded only marginal performance gains, highlighting the unique benefit of WHE's structured expansion.
- Comparison to Activation Functions: WHE outperformed other methods that increase non-linearity solely through activation function design.
Limitations and Future Work
- Weight Decay Sensitivity: The paper notes that the WHE layer can be sensitive to weight decay settings, with a fixed, high weight decay potentially causing parameters to become zero. They propose an "adjustable weight decay policy" to mitigate this.
- Group Size Impact: While increasing the group size
gconsistently improves accuracy, there's an implicit trade-off with computational cost. The optimalgwould likely depend on the specific architecture and task. - Future Work: The authors suggest exploring different connectivity patterns within the WHE layer and investigating its application to other network architectures and tasks. They also imply that further optimization of the on-the-fly computation and memory management could yield even greater efficiency.
Summary updated