TL;DR
This paper introduces "Repaint," a novel method for data augmentation that generates diverse, yet shape-consistent, variations of training images. By repainting existing images with new textures and colors, Repaint effectively combats the "texture bias" in CNNs, leading to significant improvements in generalization for image classification and object detection tasks.
What Problem Does This Paper Address?
Deep Convolutional Neural Networks (CNNs) are powerful tools for visual tasks, but they often suffer from overfitting, a phenomenon where the model performs well on training data but poorly on unseen data. A key reason for this is the "texture bias" observed in CNNs, where they tend to rely more on easily learnable texture features than on more robust shape attributes. This bias can lead to poor generalization, especially when the training and testing data have different textural characteristics. Existing data augmentation techniques often focus on global image transformations, which may not sufficiently address this texture bias. The paper aims to improve CNN generalization by creating more diverse training examples that specifically challenge the model's reliance on texture while preserving object shapes.
Methods
The core of the Repaint method is a generative semantic synthesis model, inspired by VAE-GAN architectures. This model takes an input image and generates multiple "repainted" instances. The key innovation lies in how these instances are created:
- Shape Preservation: The model uses SPADE (Spatially Adaptive Denormalization) modules, which are crucial for ensuring that the generated images maintain the original shapes and structures of objects and regions. This is achieved by conditioning the normalization layers on semantic segmentation masks.
- Texture Diversification: While shapes are preserved, the textures and colors of the repainted images are significantly altered. This is done by sampling from a learned distribution of textures and colors, effectively creating variations that mimic different lighting conditions, seasons, or even uncovering hidden texture information.
- Task-Aware Generation: The generative model is trained in conjunction with a downstream task (e.g., image classification or object detection). This means the generated images are not just visually diverse but are also optimized to help the downstream task learn more robust features. The overall objective function includes losses for the generator, discriminator, and the downstream task itself.
Datasets
The paper evaluates Repaint on two standard large-scale datasets:
- ImageNet: Used for image classification experiments.
- Microsoft COCO: Used for object detection experiments.
The authors also mention using masks generated by DeepLab-v2 and the Felzenszwalb-Huttenlocher (FH) algorithm for semantic segmentation, highlighting that their method can work with both high-quality and approximate masks.
Key Insights and Ideas
- Texture Bias is a Real Problem: The paper reinforces the finding that CNNs are biased towards texture, which hinders generalization. Repaint directly tackles this by diversifying texture while keeping shapes consistent.
- Shape Consistency is Key for Augmentation: Unlike many style transfer methods that produce artistic effects, Repaint's focus on preserving shapes makes the generated images more suitable for downstream tasks and allows for the reuse of existing ground truth labels (e.g., bounding boxes).
- Task-Aware Generation Boosts Performance: Integrating the downstream task loss into the generative process ensures that the repainted images are not just diverse but actively help the model learn better.
- Orthogonality with Other Augmentations: Repainting can be combined with existing augmentation techniques (like CutMix or DropBlock) and regularization methods to achieve even greater performance gains.
Key Results
The Repaint method consistently improved performance across various network architectures and dataset sizes for both image classification and object detection.
- Image Classification (ImageNet): Repaint achieved an average improvement of +0.77% in Top-1 accuracy on ResNet-50 when combined with other augmentations, and showed consistent gains across different models like MobileNetv2, ResNet-50, and the EfficientNet family.
- Object Detection (COCO): Repaint led to an average improvement of +1.22 mAP on EfficientDet models.
- Low Data Regimes: The gains from Repaint were even more pronounced when using a smaller percentage of the training data (e.g., 1% or 10%), suggesting its effectiveness in data-scarce scenarios.
- Mask Quality: While DeepLab-v2 masks yielded slightly better results, Repaint still provided significant improvements even with less accurate FH masks.
Limitations and Future Work
- Fine Details: The authors acknowledge that Repaint sometimes struggles with finer details, particularly facial features, in natural, diverse datasets. This is attributed to using general-purpose datasets rather than specialized ones for face generation.
- Computational Overhead: The generative module adds computational overhead to the training process. However, this can be mitigated by compressing the module, offline training, or applying repainting to only a subset of training examples.
- Future Directions: The paper suggests exploring shape bias more thoroughly and investigating how to diversify shape attributes in a controlled manner. They also propose studying fine-grained classification tasks and evaluating the impact of texture and shape bias in such scenarios.
Summary updated