Image-Adaptive Hint Generation via Vision Transformer for Outpainting

Daehyeon Kong, Kyeongbo Kong, Kyunghun Kim, Sung-Jun Min, Suk-Ju Kang

Introduction

Image outpainting—the task of extending an image beyond its original boundaries—presents a significant challenge in computer vision. Unlike inpainting, which benefits from surrounding context on all sides, outpainting requires the model to "hallucinate" content based on information from only one side. This often leads to structural inconsistencies, repetitive patterns, or unnatural artifacts.

Traditional approaches often attempt to simplify outpainting by converting it into an inpainting task using geometric rearrangements. Common techniques include Mirrored Rearrangement (MR), which mirrors edge pixels, and Bidirectional Rearrangement (BR), which swaps the left and right sides of the image. While effective for simple, symmetric textures, these methods fail on complex, asymmetric scenes like shorelines or urban landscapes, frequently producing "V-shaped" artifacts or breaking the scene's global logic.

Methodology: The Hint-Based Framework

The proposed method introduces a two-stage "hint-guided" framework that leverages the global attention capabilities of Vision Transformers (ViT) to provide structural anchors for outpainting.

1. Hint Generation via Vision Transformer

Instead of relying on local edge pixels, the model uses a ViT to analyze the entire input image. Through self-attention, the ViT identifies patches from anywhere in the image that are contextually relevant to the missing regions. It then generates small "hint" strips that are attached to the boundaries of the input image. These hints act as a target for the subsequent inpainting stage, effectively turning an open-ended outpainting problem into a constrained inpainting task.

Comparison of rearrangement strategies: Bidirectional, Mirrored, and the proposed Hint-based method. The Hint-based approach avoids the "V-shaped" artifacts common in geometric rearrangements.

2. Inpainting and Post-processing

With the hints in place, the gap between the original image and the hints is treated as a hole to be filled. The framework is model-agnostic, meaning it can use standard off-the-shelf inpainting networks such as RFR-Net or EdgeConnect. Once the inpainting network fills the missing regions, the synthetic hints are cropped out, resulting in a seamless extension of the original scene.

Experimental Evaluation

The researchers evaluated the framework on the Beach and SUN datasets, which feature diverse and often asymmetric natural scenes.

Quantitative and Qualitative Results

The hint-based method consistently outperformed baselines across multiple metrics. On the Beach dataset using RFR-Net, the proposed method achieved a Fréchet Inception Distance (FID) of 31.81, a significant improvement over the baseline outpainting (36.00) and Mirrored Rearrangement (36.65). A lower FID score indicates that the generated images are more realistic and diverse.

Visual analysis confirms that the hint-based approach maintains structural coherence better than previous methods. It successfully continues complex elements like slanted shorelines or specific objects without creating unnatural symmetry.

Qualitative results across various scenes. The "Hint" column shows the generated strips, while "RFR-Net (w/ Hint)" demonstrates the final outpainted result compared to the ground truth and the no-hint baseline.

In a Mean Opinion Score (MOS) study involving 30 raters, the proposed method received a score of 4.74/5.0, vastly outperforming Mirrored (3.12) and Bidirectional (2.98) rearrangements, highlighting its superior perceptual quality.

Conclusion

By reframing outpainting as a hint-guided inpainting task, this research provides a robust solution for extending complex, asymmetric images. The use of a Vision Transformer for adaptive hint generation allows the model to capture global context, ensuring that the extended regions are structurally and logically consistent with the original content. While the two-stage process adds some computational overhead, its compatibility with existing inpainting architectures makes it a versatile tool for high-quality image synthesis.

Summary updated

Abstract

Image outpainting has recently received considerable attention because it can be useful in tasks such as image retargeting and panorama image generation. In general, the problem of extending an image beyond its given boundaries is still ill-posed. Conventional methods predominantly attempt image outpainting by using complex network structures. Some recent studies have tried to decrease the problem complexity through the conversion techniques from outpainting to inpainting. Although these methodologies work well in simple cases, their performance reduces considerably for asymmetrical images. This paper proposes a novel hint-based outpainting methodology that can adaptively select the most plausible patches as hints from a given image to reduce the difficulty of outpainting. To estimate high-quality hints, inspired by patch-based image inpainting methods, we utilize Vision Transformer that considers self-attention for each patch. The estimated hints are attached on both boundaries of the input image and the inside missing regions are predicted by using an inpainting network. After finishing the prediction, the output image is obtained by removing the hints. Experiments show that our image-adaptive hint framework, when employed in representative inpainting networks, can consistently improve its performance compared to the other conversion techniques from outpainting to inpainting on SUN and Beach benchmark datasets.
BibTeX
@inproceedings{kong2022image,
  author = {Daehyeon Kong and Kyeongbo Kong and Kyunghun Kim and Sung-Jun Min and Suk-Ju Kang},
  title = {{Image-Adaptive Hint Generation via Vision Transformer for Outpainting}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
  doi = {10.1109/wacv51458.2022.00408},
}
Figure