Block Shuffle: A Method for High-Resolution Fast Style Transfer With Limited Memory

Weifeng Ma, Zhe Chen, Caoting JiIEEE Access 2020 · DOI

TL;DR

This paper introduces "Block Shuffle," a novel method to enable high-resolution style transfer on devices with limited memory. By breaking down large images into smaller, manageable blocks, shuffling them, and then reassembling them, Block Shuffle overcomes out-of-memory errors while producing visually coherent results that surpass existing feathering-based methods.

What Problem Does This Paper Address?

Neural Style Transfer (NST) allows users to render images in the artistic style of another. "Fast Style Transfer" (FST) uses feed-forward neural networks to achieve this much faster than original methods. However, FST struggles with high-resolution images due to the significant memory requirements of these networks, leading to "out-of-memory" (OOM) errors on most mobile devices and personal computers. Existing solutions either involve costly hardware upgrades or employ "feathering-based" methods that split images into overlapping tiles and stitch them together. While these feathering methods avoid OOM errors, they often result in noticeable seamlines and reduced image quality. This paper aims to bridge this gap by enabling high-resolution FST on memory-constrained devices without compromising visual quality.

Methods

The core of the proposed "Block Shuffle" method lies in its innovative pre-processing and post-processing steps that work around the memory limitations of FST models.

Pre-processing: 1. Expand: The input image is padded to ensure that when it's divided into blocks, the basic regions of adjacent blocks can overlap. 2. Cut: The expanded image is then cut into overlapping square blocks of a defined W_block size. These blocks are numbered sequentially. 3. Shuffle: The list of image blocks is randomly shuffled. This is a key step that helps break up spatial correlations that might lead to artifacts. 4. Concatenate: The shuffled blocks are concatenated into smaller "sub-images." The size of these sub-images is constrained by the maximum resolution the FST model can handle (e.g., W_max x W_max). This step effectively breaks down a large, memory-intensive task into multiple smaller, manageable tasks.

Style Transfer: The FST model is then applied to each of these sub-images.

Post-processing: 1. Recut: The stylized sub-images are cut back into blocks of W_block size. A small border (8 pixels) is removed from each block to mitigate boundary effects from the style transfer process. 2. Sort: The blocks are sorted back into their original sequence based on their numbering. 3. Restore: The sorted blocks are stitched together. A feathering effect is used to blend the overlapping regions of adjacent blocks, creating a smoother transition. 4. Smooth: Finally, bilateral filters are applied to the entire image to eliminate any remaining seamlines and small noise textures, further enhancing visual coherence.

The method is designed to be non-invasive, meaning it can be used as a plug-in for existing FST models without requiring retraining.

Datasets

  • MS-COCO dataset: Used for training the FST network.
  • The paper uses standard style and content images for evaluation, with resolutions up to 10000x10000 pixels.

Key Insights and Ideas

  • Decomposition as a Memory Solution: The fundamental insight is that a large, memory-intensive task (high-resolution style transfer) can be decomposed into smaller, independent sub-tasks (stylizing smaller blocks) that fit within memory constraints.
  • Shuffling for Coherence: Randomly shuffling the image blocks before stylizing and then sorting them back is crucial. This disrupts the direct spatial relationship between adjacent blocks during stylization, which, combined with the post-processing, helps to create a more globally coherent and less artifact-prone result than simple concatenation or feathering.
  • Pixel Distribution Matching: The paper hypothesizes that if the pixel distribution of the sub-images closely matches that of the original input image, the stylized results will be more easily stitched together. Their "pixel distribution matching" method (a precursor to Block Shuffle) aims to achieve this by selecting blocks that better represent the input's overall distribution.
  • Non-invasive Approach: The method's ability to act as a plug-in for existing FST models without retraining is a significant practical advantage.

Key Results

  • Enables High-Resolution Transfer: Block Shuffle successfully enables style transfer for images up to 10000x10000 pixels on devices with limited memory, a feat impossible for the baseline FST method.
  • Improved Visual Quality: Compared to the feathering-based method, Block Shuffle significantly reduces seamlines and noise textures, resulting in higher quality stylized images that are more visually coherent.
  • Speed vs. Memory Trade-off: While Block Shuffle is an order of magnitude slower than the baseline FST method, it offers a crucial trade-off: it breaks the memory barrier, making high-resolution style transfer accessible on a wider range of devices. For example, on a mobile phone, stylizing a 1000x1000 image takes 16.21 seconds with Block Shuffle, compared to 1.23 seconds for the baseline (which would fail on larger images).
  • Memory Reduction: The memory usage of Block Shuffle is significantly lower than the baseline for high-resolution images, allowing devices to process much larger inputs without OOM errors.

Limitations and Future Work

  • Speed: The primary limitation acknowledged is the significant increase in processing time compared to the baseline FST method. The authors state their method is an "order of magnitude slower."
  • Future Work: The authors suggest further research to improve image quality and speed. They also aim to promote the industrialization of Fast Style Transfer by making high-resolution stylization more accessible.

Summary updated

BibTeX
@article{ma2020block,
  author = {Weifeng Ma and Zhe Chen and Caoting Ji},
  title = {{Block Shuffle: A Method for High-Resolution Fast Style Transfer With Limited Memory}},
  journal = {IEEE Access},
  year = {2020},
  doi = {10.1109/access.2020.3020053},
}
Figure