Meta-Learning and Self-Supervised Pretraining for Real World Image Translation

Ileana Rugina, Rumen Dangovski, Mark Veillette, Pooya Khorrami, Brian Cheung, Olga Simek, Marin SoljacicarXiv 2021 · arXiv

Abstract

This paper introduces a new benchmark for few-shot image-to-image translation using the SEVIR weather dataset, moving beyond saturated "toy" benchmarks. By applying Model-Agnostic Meta-Learning (MAML) and self-supervised contrastive pretraining (MoCoV3) with novel "natural" temporal augmentations, we demonstrate significant improvements in generating realistic, high-intensity radar imagery from sparse satellite data.

Introduction

While deep learning has revolutionized computer vision, its success often hinges on massive, human-curated datasets. In real-world applications like meteorology or medical imaging, data is frequently sparse, expensive to label, and subject to distributional shifts. Existing few-shot learning benchmarks, such as Omniglot or MiniImageNet, often rely on simple icons that fail to capture the complexity of physical phenomena. Furthermore, most few-shot research focuses on classification, leaving image-to-image translation relatively unexplored. This study tackles the challenge of few-shot image translation: learning to generate complex radar imagery (VIL) from satellite infrared (IR) and lightning data using only a handful of examples from a specific weather event.

Methodology

To address the low-data regime, we propose a multi-pronged approach combining meta-learning and self-supervision.

Meta-Learning for GANs

We adapt Model-Agnostic Meta-Learning (MAML) to a Generative Adversarial Network (GAN) framework. The objective is to find a weight initialization for a U-Net generator and a patch discriminator that can rapidly adapt to a new task—a specific storm event—with minimal gradient steps. This involves a "bilevel" optimization process where the model learns an optimal starting point for adaptation.

Self-Supervised Pretraining and Natural Augmentations

We utilize MoCoV3, a contrastive learning method, to pretrain the U-Net's encoder. Recognizing that standard augmentations like color jitter are inappropriate for physical sensor data, we exploit the spatio-temporal structure of weather data. We use adjacent frames in a time series as "natural" augmentations, teaching the model representation invariance over short time intervals by anchoring one frame as a query and a neighboring frame as a positive key.

Input modalities and target VIL from the SEVIR dataset.

Experimental Setup

The study utilizes the Storm Event Imagery (SEVIR) dataset, which contains over 10,000 weather events across five sensor modalities. The task is to translate IR and Lightning data into Vertically Integrated Liquid (VIL) imagery. Each task corresponds to a single storm event, where the first 10 frames serve as the support set for training and the subsequent 10 frames serve as the query set for evaluation.

Results and Discussion

Our evaluation revealed several key insights into the performance of meta-learning in generative tasks.

Quantitative and Qualitative Performance

MAML-based models consistently outperformed standard joint training in minimizing Mean Absolute Error (MAE). Qualitatively, MAML generators were significantly better at identifying and "filling in" sparse, high-intensity VIL regions that joint training models often ignored or blurred. Contrastive pretraining with "Level 4" augmentations (combining temporal shifts with spatial transforms) showed consistent improvements across meteorological metrics, particularly the Critical Success Index (CSI) and Probability of Detection (POD) at high thresholds.

Qualitative comparison between the target VIL, joint training prediction, and MAML prediction.

The Precision-Recall Trade-off

An interesting observation was the trade-off between optimization objectives and domain-specific metrics. While MAML achieved lower reconstruction error, it sometimes resulted in a lower Probability of Detection (recall) compared to joint training. Meta-learned models tended to be more "cautious," accurately identifying storm centers (high precision) but potentially missing peripheral, lighter precipitation.

Conclusion

This work establishes a foundation for few-shot generative modeling on complex, non-RGB data. While MAML shows great promise for capturing high-intensity features, training stability remains a challenge due to the interplay between adversarial loss and bilevel optimization. Future work will explore techniques like spectral normalization to improve stability and investigate better evaluation metrics that correlate more strongly with visual realism in meteorological contexts.

Summary updated

Abstract

Recent advances in deep learning, in particular enabled by hardware advances and big data, have provided impressive results across a wide range of computational problems such as computer vision, natural language, or reinforcement learning. Many of these improvements are however constrained to problems with large-scale curated data-sets which require a lot of human labor to gather. Additionally, these models tend to generalize poorly under both slight distributional shifts and low-data regimes. In recent years, emerging fields such as meta-learning or self-supervised learning have been closing the gap between proof-of-concept results and real-life applications of machine learning by extending deep-learning to the semi-supervised and few-shot domains. We follow this line of work and explore spatio-temporal structure in a recently introduced image-to-image translation problem in order to: i) formulate a novel multi-task few-shot image generation benchmark and ii) explore data augmentations in contrastive pre-training for image translation downstream tasks. We present several baselines for the few-shot problem and discuss trade-offs between different approaches. Our code is available at https://github.com/irugina/meta-image-translation.
BibTeX
@misc{rugina2021meta,
  author = {Ileana Rugina and Rumen Dangovski and Mark Veillette and Pooya Khorrami and Brian Cheung and Olga Simek and Marin Soljacic},
  title = {{Meta-Learning and Self-Supervised Pretraining for Real World Image Translation}},
  year = {2021},
  eprint = {2112.11929},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2112.11929},
}
Figure