Distribution Estimation to Automate Transformation Policies for Self-Supervision

Seunghan Yang, Debasmit Das, Simyung Chang, Sungrack Yun, Fatih PorikliarXiv 2021 · arXiv

Introduction

Self-supervised learning (SSL) has revolutionized representation learning by allowing models to learn from unlabeled data through "pretext tasks." A common approach involves applying visual transformations—such as rotation, scaling, or color jittering—and training a model to predict the parameters of those transformations. However, the effectiveness of these tasks is governed by the Visual Transformation for Self-Supervision (VTSS) hypothesis: if a transformation is already inherent in the dataset (e.g., rotated images in satellite data), the pretext task becomes trivial, failing to force the model to learn meaningful features.

Historically, identifying these "conflicting" transformations required manual inspection and domain expertise. This paper introduces an automated framework that leverages Generative Adversarial Networks (GANs) to mathematically estimate the distribution of transformations within a dataset and automatically derive optimal self-supervision policies.

The Automated Framework

The proposed method automates the selection of pretext tasks through a two-stage learning process.

1. Distribution Estimation via GANs

The core of the system is an adversarial setup designed to "discover" existing transformations. A Generator produces transformation parameters (e.g., rotation angles or scale factors) via a mapping network. A Discriminator is then tasked with distinguishing between original images from the target dataset ($x_c$) and images from a "reference subset" ($x_{ref}$) that have been modified by the Generator's sampled transformations ($t_\theta$).

Overview of the GAN-based framework for distribution estimation (a) and the mapping network for transformation parameters (b).

As training progresses, the Generator learns to produce transformations that make the reference images indistinguishable from the target dataset. The resulting distribution ($p_\theta$) represents the transformations already present in the data.

2. Automated Transformation Policy (ATP)

Once the existing distribution is identified, the framework calculates its complementary distribution ($q_\theta$). This distribution represents transformations that are rare or missing in the dataset. By sampling from $q_\theta$ to create pretext tasks, the system ensures that the self-supervised model is challenged by novel variations, preventing "transformation conflict" and encouraging the learning of robust, high-level features.

Key Insights

  • Adversarial Discovery: The "aha" moment of this research is the realization that if a Generator can fool a Discriminator by transforming a standard image, that specific transformation must be a defining characteristic of the target dataset.
  • Artifact Mitigation: Transformations like rotation can introduce black corners or zero-padding, which act as "shortcuts" for the Discriminator. The authors implemented center-cropping post-transformation to ensure the model focuses on semantic content rather than geometric artifacts.
  • Targeted vs. Random: The results suggest that "targeted" self-supervision—specifically avoiding what the data already contains—is significantly more effective than applying random augmentations.

Results and Discussion

The researchers evaluated the Automated Transformation Policy (ATP) across diverse benchmarks, including standard vision datasets (MNIST, CIFAR) and specialized domains like medical imaging (ChestX) and satellite imagery (EuroSAT).

Estimated transformation distributions for SVHN (top) and corresponding downstream accuracy for various scale and translation tasks (bottom).

On the SVHN dataset, the impact of the automated policy was particularly pronounced. When the model was trained on scaling transformations already prevalent in the data, downstream supervised accuracy was only 54.30%. By switching to the complementary distribution identified by the GAN, accuracy rose to 67.03%. Similarly, on Fashion MNIST, the automated policy achieved 93.22% accuracy, outperforming standard baselines like RotNet (91.94%).

Conclusion and Future Work

By automating the identification of inherent data transformations, this framework removes a significant manual bottleneck in the SSL pipeline. While the current study focused on small-to-medium datasets, the results demonstrate that ATP consistently outperforms both random policies and those based on human intuition. Future work will involve scaling this approach to high-dimensional, large-scale datasets like ImageNet, where manual transformation analysis is increasingly impractical.

Summary updated

Abstract

In recent visual self-supervision works, an imitated classification objective, called pretext task, is established by assigning labels to transformed or augmented input images. The goal of pretext can be predicting what transformations are applied to the image. However, it is observed that image transformations already present in the dataset might be less effective in learning such self-supervised representations. Building on this observation, we propose a framework based on generative adversarial network to automatically find the transformations which are not present in the input dataset and thus effective for the self-supervised learning. This automated policy allows to estimate the transformation distribution of a dataset and also construct its complementary distribution from which training pairs are sampled for the pretext task. We evaluated our framework using several visual recognition datasets to show the efficacy of our automated transformation policy.
BibTeX
@misc{yang2021distribution,
  author = {Seunghan Yang and Debasmit Das and Simyung Chang and Sungrack Yun and Fatih Porikli},
  title = {{Distribution Estimation to Automate Transformation Policies for Self-Supervision}},
  year = {2021},
  eprint = {2111.12265},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2111.12265},
}
Figure