Learning Image Representations by Completing Damaged Jigsaw Puzzles

Dahun Kim, Donghyeon Cho, Donggeun Yoo, In So Kweon

TL;DR

This paper proposes a novel self-supervised learning approach by "damaging" existing tasks like jigsaw puzzles, inpainting, and colorization to make them more challenging. Their core idea is that forcing a network to recover from more severe damage leads to more robust and generalizable image representations, achieving state-of-the-art results on PASCAL classification and semantic segmentation.

What Problem Does This Paper Address?

Learning powerful visual representations is crucial for computer vision tasks, but traditional methods often rely on massive amounts of human-labeled data. This is expensive and limits scalability. Self-supervised learning offers a way around this by creating supervisory signals directly from unlabeled data. However, existing self-supervised tasks, while effective, might not push networks hard enough to learn truly general-purpose features. The authors aim to address this by increasing the difficulty of these tasks, hypothesizing that this will force networks to learn richer, more transferable representations.

Methods

The paper introduces two main methodological contributions:

  1. Complicating Existing Self-Supervised Tasks: They take three popular self-supervised tasks – jigsaw puzzles, inpainting, and colorization – and make them more challenging:
  2. Jigsaw Puzzles: In addition to shuffling patches, they introduce channel-wise damage (dropping L or ab channels from different patches) and remove one puzzle piece entirely.
  3. Inpainting: Instead of restoring a missing patch in all channels, they require the network to generate the color (ab channels) of a missing region using only the grayscale (L channel) of the surrounding pixels.
  4. Colorization: They reduce the input to only one quarter of the image (a single patch's L channel) and ask the network to colorize it.
  5. "Completing Damaged Jigsaw Puzzles" (CDJP): This is their novel, overarching task that simultaneously combines the complicated versions of jigsaw puzzles, inpainting, and colorization. The network must arrange shuffled patches, recover a missing patch, and colorize the entire image, all while dealing with the aforementioned damages. This is implemented as a 9-tower siamese network with a shared AlexNet backbone and task-specific branches.

Datasets

  • Training: ImageNet (1.3 million images) was used for pre-training the network without any labels.
  • Evaluation: PASCAL VOC 2007 and PASCAL VOC 2012 datasets were used for transfer learning experiments on classification, detection, and semantic segmentation.

Key Insights and Ideas

  • Difficulty Drives Robustness: The core insight is that increasing the difficulty of self-supervised tasks forces networks to learn more robust and generalizable representations. Simply making tasks harder, rather than introducing entirely new paradigms, can be highly effective.
  • Synergy of Tasks: Combining multiple, even complicated, self-supervised tasks simultaneously leads to better representations than any single task alone. The CDJP task leverages the complementary knowledge learned from spatial configuration, image context, and cross-channel relations.
  • Task-Specific Damage is Key: The specific ways they complicate each task (e.g., cross-channel inpainting, partial context for colorization) are designed to force the network to learn specific types of information (contextual, cross-channel relations) that are crucial for general visual understanding.

Key Results

  • Complicated Tasks Outperform Originals: In Section 5.1, complicating each individual task (jigsaw, inpainting, colorization) consistently improved performance on PASCAL classification and segmentation compared to their original versions. For example, complicated colorization achieved 66.7% classification mAP, compared to 65.9% for standard colorization.
  • CDJP Achieves State-of-the-Art: The proposed "Completing Damaged Jigsaw Puzzles" (CDJP) method achieved state-of-the-art performance on PASCAL classification (69.2% mAP) and semantic segmentation (39.3% mIoU), outperforming previous self-supervised methods and even simple combinations of tasks.
  • Strong Linear Classification: On ImageNet linear classification, CDJP's learned features (especially in higher layers like conv4 and conv5) achieved competitive, near state-of-the-art performance, demonstrating good generalizability.
  • Qualitative Improvements: Nearest neighbor search experiments showed that CDJP's representations are more invariant to pose/viewpoint variations and more robust to intra-class color variations compared to single-task baselines, aligning better with ImageNet pretraining.

Limitations and Future Work

  • Computational Cost: While not explicitly detailed as a limitation, training a 9-tower siamese network with complicated tasks is likely computationally intensive.
  • Transfer to Other Tasks: The paper mentions that further transfer learning results on tasks like depth prediction and with deeper networks (like VGG) are presented in supplementary material, suggesting these were explored but not the primary focus of the main paper.
  • Hyperparameter Tuning: The paper uses fixed weights ($\alpha, \beta = 0.01$) for combining losses in CDJP. Further exploration of these weights could potentially yield even better results.
  • Future Work: The authors suggest that their approach of complicating self-supervised tasks can be extended to other tasks and domains. They also highlight the potential for their method to be used with deeper architectures.

Summary updated

Abstract

In this paper, we explore methods of complicating selfsupervised tasks for representation learning. That is, we do severe damage to data and encourage a network to recover them. First, we complicate each of three powerful self-supervised task candidates: jigsaw puzzle, inpainting, and colorization. In addition, we introduce a novel complicated self-supervised task called "Completing damaged jigsaw puzzles" which is puzzles with one piece missing and the other pieces without color. We train a convolutional neural network not only to solve the puzzles, but also generate the missing content and colorize the puzzles. The recovery of the aforementioned damage pushes the network to obtain robust and general-purpose representations. We demonstrate that complicating the self-supervised tasks improves their original versions and that our final task learns more robust and transferable representations compared to the previous methods, as well as the simple combination of our candidate tasks. Our approach achieves state-of-the-art performance in transfer learning on PASCAL classification and semantic segmentation.
BibTeX
@inproceedings{kim2018learning,
  author = {Dahun Kim and Donghyeon Cho and Donggeun Yoo and In So Kweon},
  title = {{Learning Image Representations by Completing Damaged Jigsaw Puzzles}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2018},
  url = {https://openreview.net/forum?id=ESe16bm_CSs},
}
Figure