Introduction
Image-to-image (I2I) translation tasks, such as day-to-night conversion or haze removal, are typically performed using Fully Convolutional Networks (FCNs). However, FCNs suffer from a significant scaling bottleneck: their computational cost increases linearly with the number of pixels. Processing high-resolution images (e.g., 4K) with standard architectures like CycleGAN requires approximately 7.2 Tera-multiply-accumulate operations (MACs), making real-time performance impossible on current hardware.
To address this, researchers often design lightweight models from scratch, which frequently compromises image quality. This paper introduces the Multi-Curve Translator (MCT), a plug-in module that enables existing FCNs to process high-resolution images efficiently by shifting computation from pixel space to a low-resolution parameter space.
The Multi-Curve Translator Architecture
The core innovation of MCT is its shift from predicting pixels to predicting the parameters of local mapping curves. Instead of building a new model, MCT utilizes an existing FCN as a "backbone" and modifies only its final layer.
Key Components:
- Parameter Map Prediction: The backbone receives a downsampled version of the input image. It outputs a "parameter map" where each entry defines a set of 1D Look-Up Tables (LUTs) or curves for a local region.
- Multi-Curve Translation: To capture complex color interactions, MCT employs a channel-crossing strategy, learning 9 curves per region to map between R, G, and B channels (e.g., R-to-R, R-to-G, R-to-B).
- Slicing Operation: Using trilinear interpolation (inspired by bilateral grids), the predicted curves are applied to the original high-resolution input. This allows high-frequency details to bypass the FCN bottleneck, flowing directly from the input to the output.
Training Strategies
To ensure stable training and high-quality results, the authors implement two specific strategies:
- Base Output Constraint: The network is forced to learn a "base" translation alongside residual curve adjustments.
- Pixel Non-Alignment: A data augmentation technique involving padding and random cropping that prevents the network from relying solely on spatial alignment, forcing it to extract more robust high-frequency information.
Experimental Results
The MCT module was evaluated across tasks including style transfer, dehazing, and photo retouching using datasets like Flickr (480p to 8K), Microsoft COCO, and MIT-Adobe-5K.
Performance and Quality:
- Efficiency: MCT-CycleGAN achieves a 40x speedup over the original CycleGAN, processing 4K images at over 30 FPS on an NVIDIA RTX 3090.
- Quantitative Gains: In dehazing benchmarks (HazeRD), MCT-GCANet achieved a PSNR of 17.19, outperforming the base model's 16.69.
- Detail Preservation: Because the high-resolution input serves as the coordinate for LUT lookups, the output retains sharpness that standard FCNs often lose at high resolutions. In user studies, 67.4% of respondents preferred MCT-CycleGAN for its superior detail retention.
Conclusion and Limitations
MCT provides a powerful, efficient way to scale I2I models to 4K resolution. However, its reliance on local color and tone mapping means it is restricted to photorealistic translations. It cannot perform tasks requiring significant structural changes, such as shape deformation (e.g., horse2zebra). Future work may explore integrating model compression or extending the architecture to handle non-smooth spatial transformations.
Summary updated