Review Neural Networks about Image Transformation Based on IGC Learning Framework with Annotated Information

Yuanjie Yan, Suorong Yang, Yan Wang, Jian Zhao, Furao ShenarXiv 2022 · OpenReview

Abstract

Image transformation, a class of vision and graphics problems whose goal is to learn the mapping between an input image and an output image, develops rapidly in the context of deep neural networks. In Computer Vision (CV), many problems can be regarded as the image transformation task, e.g., semantic segmentation and style transfer. This paper proposes a novel learning framework including Independent learning, Guided learning, and Cooperative learning, called the IGC learning framework. We review those subtasks and give a unified interpretation of various scenarios, categorizing related subtasks according to similar development trends. Experiments verify the effectiveness of IGC learning.

Introduction

In the rapidly evolving world of Computer Vision (CV), tasks like turning a sketch into a photo, swapping a summer landscape for winter, or applying Van Gogh’s style to a selfie often feel like distinct problems. However, a recent paper by Yuanjie Yan and colleagues argues that these are all branches of the same tree. By introducing the IGC (Independent, Guided, and Cooperative) learning framework, the authors provide a unified lens to understand how neural networks transform images based on the information available to them.

The field of image transformation is currently fragmented, with researchers often treating "Style Transfer" (artistic) and "Image-to-Image Translation" (functional) as separate domains. The IGC framework identifies a lack of a unified structure that explains how these models utilize annotated information—whether that information comes from manual human labels, pre-trained "teacher" networks, or competing "student" networks.

The IGC Learning Framework

The core contribution of this research is the classification of transformation models into three distinct paradigms:

  1. Independent Learning (I): The standard approach where a student model learns directly from a labeled dataset. Examples include FCN and U-Net architectures used for semantic segmentation.
  2. Guided Learning (G): A student model learns from a pre-trained Teacher model. This is the backbone of Neural Style Transfer (NST), where a pre-trained network (like VGG) guides the transformation of a content image into a specific artistic style.
  3. Cooperative Learning (C): Multiple student models interact to improve each other. The most prominent example is the Generative Adversarial Network (GAN), where a Generator and a Discriminator "cooperate" through competition to produce realistic images. This also includes CycleGAN, where two generators work in a loop to ensure transformations are reversible.
Examples of image style transformation using the IGC framework, showing (A) original image and (B-D) various style transfer results.

Key Insights and Results

The "Annotation" Shift

The authors argue that "labels" are not limited to manual tags. In Guided learning, the features extracted by a teacher model act as the annotation. In Cooperative learning, the output of one model serves as the label for another, shifting the definition of supervised learning in the context of image generation.

Texture vs. Structure

A key finding is the comparison between Independent and Cooperative learning. Independent models (like U-Net) often produce blurred results because they optimize for pixel-perfect accuracy. Cooperative models (like Pix2Pix) produce sharper, more realistic textures because the "Discriminator" student forces the "Generator" student to prioritize visual realism over simple pixel-matching. Experiments on the Facades dataset showed that while U-Net had lower per-pixel loss, GAN-based cooperation added necessary vividness.

Cycle Consistency and Efficiency

For unaligned data (like Apple2Orange), the "Cycle Consistency" loss is a form of cooperative learning that prevents the model from ignoring the input image. Furthermore, the paper highlights the evolution from iterative methods, which take minutes per image, to feed-forward methods that can transform images in real-time.

Discussion and Future Directions

While the IGC framework provides a roadmap for current research, several open problems remain:

  • The Evaluation Gap: There is still a lack of perfect mathematical metrics for "realism." While IoU works for segmentation, it fails to capture the success of artistic style transfer.
  • Deepfake Detection: The IGC framework could be extended to detection models, where a detector acts as a student learning to identify the "fingerprints" of specific transformation paradigms.
  • Shape Transformation: Current unaligned models excel at changing colors and textures but struggle with significant shape changes (e.g., turning a cat into a dog), representing a major hurdle for the Cooperative paradigm.

Summary updated

Abstract

Image transformation, a class of vision and graphics problems whose goal is to learn the mapping between an input image and an output image, develops rapidly in the context of deep neural networks. In Computer Vision (CV), many problems can be regarded as the image transformation task, e.g., semantic segmentation and style transfer. These works have different topics and motivations, making the image transformation task flourishing. Some surveys only review the research on style transfer or image-to-image translation, all of which are just a branch of image transformation. However, none of the surveys summarize those works together in a unified framework to our best knowledge. This paper proposes a novel learning framework including Independent learning, Guided learning, and Cooperative learning, called the IGC learning framework. The image transformation we discuss mainly involves the general image-to-image translation and style transfer about deep neural networks. From the perspective of this framework, we review those subtasks and give a unified interpretation of various scenarios. We categorize related subtasks about the image transformation according to similar development trends. Furthermore, experiments have been performed to verify the effectiveness of IGC learning. Finally, new research directions and open problems are discussed for future research.
BibTeX
@misc{yan2022review,
  author = {Yuanjie Yan and Suorong Yang and Yan Wang and Jian Zhao and Furao Shen},
  title = {{Review Neural Networks about Image Transformation Based on IGC Learning Framework with Annotated Information}},
  year = {2022},
  eprint = {2206.10155},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2206.10155},
}
Figure