Language-based Photo Color Adjustment for Graphic Designs

Zhenwei Wang, Nanxuan Zhao, Gerhard P. Hancke, Rynson W. H. Lau

Abstract

Adjusting the photo color to associate with some design elements is an essential way for a graphic design to effectively deliver its message and make it aesthetically pleasing. However, existing tools and previous works face a dilemma between the ease of use and level of expressiveness. To this end, we introduce an interactive language-based approach for photo recoloring, which provides an intuitive system that can assist both experts and novices on graphic design. Given a graphic design containing a photo that needs to be recolored, our model can predict the source colors and the target regions, and then recolor the target regions with the source colors based on the given language-based instruction. The multi-granularity of the instruction allows diverse user intentions. The proposed novel task faces several unique challenges, including: 1) color accuracy for recoloring with exactly the same color from the target design element as specified by the user; 2) multi-granularity instructions for parsing instructions correctly to generate a specific result or multiple plausible ones; and 3) locality for recoloring in semantically meaningful local regions to preserve original image semantics. To address these challenges, we propose a model called LangRecol with two main components: the language-based source color prediction module and the semantic-palette-based photo recoloring module. We also introduce an approach for generating a synthetic graphic design dataset with instructions to enable model training. We evaluate our model via extensive experiments and user studies. We also discuss several practical applications, showing the effectiveness and practicality of our approach.

Introduction

In professional graphic design, harmonizing the colors of an image with surrounding elements—such as backgrounds, logos, or typography—is vital for visual coherence. However, users currently face a "dilemma of expressiveness." Professional tools like Adobe Photoshop offer high control but require significant expertise and time. Conversely, existing language-based AI editors often lack the precision needed for design, frequently failing in color accuracy (matching an exact hex code from a logo) or locality (accidentally recoloring unintended regions).

LangRecol handles multi-granularity instructions to recolor specific photo elements based on design context.

We introduce LangRecol, a system designed to handle "multi-granularity" instructions. It allows users to provide either broad requests (e.g., "match the background") or highly specific ones (e.g., "recolor the shirt to the color of the yellow shape"), ensuring professional-grade results with minimal effort.

Methodology

The LangRecol framework consists of two primary modules that work in tandem to interpret instructions and apply color transformations.

The LangRecol architecture: Language-based source color prediction followed by semantic-palette-based photo recoloring.

1. Language-based Source Color Prediction

This module identifies the intended source color from the graphic design. It employs a Granularity Recognition subnet to distinguish between broad and specific instructions. To ensure accuracy, the model uses Class-aware Voting: max voting for sparse elements like text to find the dominant character color, and sum voting for dense elements like shapes.

2. Semantic-palette-based Photo Recoloring

Once the target color is identified, the system applies it to the photo while maintaining realism.

  • Region Refinement: Instead of binary masks, the model generates "soft" semantic color layers. This avoids jagged edges and respects complex object boundaries.
  • Lightness Preservation: By operating in the CIE Lab color space and preserving the L-channel (lightness), the model ensures that original shading, textures, and gradients remain intact, preventing a "painted-on" look.

Evaluation and Results

To train and validate LangRecol, we developed a Synthetic Graphic Design Dataset featuring 16 types of design elements arranged according to professional layout principles. We also tested the model on a collection of 71 real-world designs from Canva.

Quantitatively, LangRecol significantly outperformed state-of-the-art baselines like ManiGAN and Open-Edit, achieving a PSNR of 24.27 and SSIM of 0.8500. In user studies, participants preferred LangRecol’s results over 80% of the time. Most notably, novice users completed design tasks 63.7% faster than when using traditional manual interfaces, saving nearly 45 seconds per design.

Conclusion

LangRecol demonstrates that combining deep learning for semantic understanding with palette-based methods for color manipulation can bridge the gap between ease of use and professional expressiveness. While future work will focus on improving robustness for decorative fonts and global scene consistency (e.g., updating reflections), LangRecol currently offers a powerful tool for streamlining the graphic design workflow.

For more details, code, and data, visit zhenwwang.github.io/langrecol.

Summary updated

Abstract

Adjusting the photo color to associate with some design elements is an essential way for a graphic design to effectively deliver its message and make it aesthetically pleasing. However, existing tools and previous works face a dilemma between the ease of use and level of expressiveness. To this end, we introduce an interactive language-based approach for photo recoloring, which provides an intuitive system that can assist both experts and novices on graphic design. Given a graphic design containing a photo that needs to be recolored, our model can predict the source colors and the target regions, and then recolor the target regions with the source colors based on the given language-based instruction. The multi-granularity of the instruction allows diverse user intentions. The proposed novel task faces several unique challenges, including: 1) color accuracy for recoloring with exactly the same color from the target design element as specified by the user; 2) multi-granularity instructions for parsing instructions correctly to generate a specific result or multiple plausible ones; and 3) locality for recoloring in semantically meaningful local regions to preserve original image semantics. To address these challenges, we propose a model called LangRecol with two main components: the language-based source color prediction module and the semantic-palette-based photo recoloring module. We also introduce an approach for generating a synthetic graphic design dataset with instructions to enable model training. We evaluate our model via extensive experiments and user studies. We also discuss several practical applications, showing the effectiveness and practicality of our approach. Please find the code and data at https://zhenwwang.github.io/langrecol.
BibTeX
@article{wang2023language,
  author = {Zhenwei Wang and Nanxuan Zhao and Gerhard P. Hancke and Rynson W. H. Lau},
  title = {{Language-based Photo Color Adjustment for Graphic Designs}},
  journal = {ACM Trans. Graphics},
  year = {2023},
  url = {https://openreview.net/forum?id=uwhVuBTkjU},
}
Figure