Augmentation Pathways Network for Visual Recognition

Yalong Bai, Mohan Zhou, Wei Zhang, Bowen Zhou, Tao Mei

Introduction

Data augmentation is a cornerstone of modern computer vision, typically relying on "light" transformations like random crops and horizontal flips. However, "heavy" augmentations—such as GridShuffle (shuffling image tiles) or extreme color shifts—often lead to training instability or accuracy degradation. This is primarily due to the significant distribution gap created between heavily augmented training data and real-world test data. While automated search algorithms like AutoAugment can identify effective policies, they are computationally expensive and lack generalizability across different architectures. The Augmentation Pathways (AP) network addresses this by proposing a structural solution that allows models to benefit from heavy augmentation without custom-searched policies.

The Augmentation Pathways Network

The core innovation is the Augmentation Pathways (AP) network, which replaces standard convolutional layers with AP-Conv layers.

Architecture comparison between a standard convolutional network and the proposed Augmentation Pathways Network.

Multi-Path Processing and Feature Sharing

An AP-Conv layer splits feature channels into distinct paths. A "Main Pathway" processes lightly augmented images, while one or more "Augmentation Pathways" handle heavily augmented versions. These paths are not isolated; the augmentation path is designed to learn visual patterns shared between light and heavy versions (e.g., object contours), while the main path focuses on features specific to the original data distribution.

Regularization and Efficiency

To ensure the pathways capture complementary information, the authors introduce a cross-pathway orthogonal constraint ($S$). This prevents redundancy and forces the pathways to decouple their learned features. Crucially, the augmentation pathways are deactivated during inference. Only the main pathway is used, resulting in a model that is both smaller and faster than the baseline. For instance, an AP-Conv layer with a 50/50 split contains only 75% of the parameters of a standard convolutional layer.

Experimental Evaluation

The approach was evaluated on ImageNet (ILSVRC-2012) and its subsets, ImageNet$_{100}$ and ImageNet$_{20}$, to test performance under varying levels of data scarcity.

Overview of heavy augmentations (left) and the resulting performance improvements across different datasets and architectures (right).

Key Results

The AP network consistently improved performance across various backbones, including ResNet, MobileNetV2, and ConvNeXt:

  • Accuracy and Efficiency: On ResNet-50, the AP variant improved Top-1 accuracy on ImageNet by 1.78% while reducing parameters from 25.6M to 21.8M (approx. 15%).
  • Data Scarcity Gains: The benefits were most pronounced on smaller datasets. On ImageNet$_{20}$, AP-ResNet-50 outperformed the standard ResNet-50 by over 3% using RandAugment.
  • Stabilizing Architectures: Certain architectures like iResNet, which typically suffer performance drops with heavy augmentation (e.g., RandAugment), showed significant gains when integrated with the AP framework.

Conclusion and Future Directions

The Augmentation Pathways network effectively filters the "side effects" of heavy augmentation (irrelevant feature bias) while retaining the "benefits" (augmented shared patterns). While the model is highly efficient during inference, the training phase requires more memory and time due to multi-path processing. Future work may explore automating the ranking of augmentation "heaviness" and extending the framework to dense prediction tasks like object detection and semantic segmentation.

Summary updated

Abstract

Data augmentation is practically helpful for visual recognition, especially at the time of data scarcity. However, such success is only limited to quite a few light augmentations (e.g., random crop, flip). Heavy augmentations are either unstable or show adverse effects during training, owing to the big gap between the original and augmented images. This paper introduces a novel network design, noted as Augmentation Pathways (AP), to systematically stabilize training on a much wider range of augmentation policies. Notably, AP tames various heavy data augmentations and stably boosts performance without a careful selection among augmentation policies. Unlike traditional single pathway, augmented images are processed in different neural paths. The main pathway handles the light augmentations, while other pathways focus on the heavier augmentations. By interacting with multiple paths in a dependent manner, the backbone network robustly learns from shared visual patterns among augmentations, and suppresses the side effect of heavy augmentations at the same time. Furthermore, we extend AP to high-order versions for high-order scenarios, demonstrating its robustness and flexibility in practical usage. Experimental results on ImageNet demonstrate the compatibility and effectiveness on a much wider range of augmentations, while consuming fewer parameters and lower computational costs at inference time.
BibTeX
@article{bai2021augmentation,
  author = {Yalong Bai and Mohan Zhou and Wei Zhang and Bowen Zhou and Tao Mei},
  title = {{Augmentation Pathways Network for Visual Recognition}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2021},
  doi = {10.1109/tpami.2023.3250330},
}
Figure