SM-CycleGAN: crop image data enhancement method based on self-attention mechanism CycleGAN

Dian Liu, Yang Cao, Jing Yang, Jianyu Wei, Jili Zhang, Chenglin Rao, Banghong Wu, Dabin ZhangScientific 2024 · DOI

Abstract

Crop disease detection and crop baking stage judgement require large image data to improve accuracy. However, the existing crop disease image datasets have high asymmetry, and the poor baking environment leads to image acquisition difficulties and colour distortion. Therefore, we explore the potential of the self-attention mechanism on crop image datasets and propose an innovative crop image data-enhancement method for recurrent generative adversarial networks (GANs) fused with the self-attention mechanism to significantly enhance the perception and information capture capabilities of recurrent GANs. By introducing the self-attention mechanism module, the cycle-consistent GAN (CycleGAN) is more adept at capturing the internal correlations and dependencies of image data, thus more effectively capturing the critical information among image data. Furthermore, we propose a new enhanced loss function for crop image data to optimise the model performance and meet specific task requirements. The experimental results show that, the peak signal-to-noise ratio of the SM-CycleGAN for tobacco images and tea leaf disease images are improved by 2.13% and 3.55%, and the structural similarity index measure is improved by 1.16% and 2.48% compared to CycleGAN, respectively.

Introduction

In modern precision agriculture, deep learning models for disease detection and maturity assessment are heavily dependent on the quality and quantity of training data. Agricultural datasets, however, frequently suffer from two major limitations. First, data scarcity and asymmetry are prevalent; collecting high-quality images of specific conditions, such as tea leaf blight, is labor-intensive, resulting in imbalanced datasets. Second, environmental distortions—particularly in tobacco roasting where high heat and humidity are present—cause significant color shifts in captured images. These distortions hinder the ability of AI systems to accurately judge leaf maturity.

While traditional data augmentation techniques like flipping or rotation are common, they do not generate new feature information. Generative Adversarial Networks (GANs), specifically CycleGAN, offer a solution by translating images between domains (e.g., from distorted to normal). However, standard CycleGANs often fail to capture long-range spatial dependencies, leading to structural inaccuracies or blurred results.

Methodology

The proposed SM-CycleGAN architecture addresses these limitations by integrating a self-attention mechanism into the standard Cycle-Consistent GAN framework.

Self-Attention Mechanism

Unlike standard convolutional layers that process images locally, the self-attention modules allow the network to "attend" to distant parts of the image simultaneously. This capability is crucial for biological structures like leaves, where global correlations—such as the relationship between the leaf tip and the stem—must be maintained to ensure realism.

The overall architecture of SM-CycleGAN, illustrating the integration of self-attention modules within the generative and discriminative cycles to capture global dependencies.

Generator and Discriminator Enhancements

The generator is modified from a standard ResBlock-based structure to include a self-attention mechanism module and an internal classifier (referred to as "sorter A" in the architecture). This classifier sits between the encoder and decoder to better distinguish between source and target domain features, improving the translation accuracy.

Detailed structure of the SM-CycleGAN generator, showcasing the placement of the self-attention mechanism module and the internal feature classifier.

Enhanced Loss Function

A specific Self-Attention Loss (SAL) is introduced and combined with traditional adversarial and cycle-consistency losses. The SAL encourages the model to focus on critical feature elements, effectively mitigating the "long-tail problem" where rare but important features are overlooked by standard models.

Experimental Results

The model was evaluated using two primary datasets: a Tobacco Roasting Dataset (206 normal and 206 distorted images) and a Tea Disease Dataset (80 normal and 80 diseased slices). SM-CycleGAN was benchmarked against models such as Pix2Pix, CycleGAN, CUT, AugCGAN, and U-GAT-IT.

Quantitative results demonstrated significant improvements over the baseline CycleGAN:

  • Tobacco Images: Peak Signal-to-Noise Ratio (PSNR) improved by 2.13% and Structural Similarity Index (SSIM) by 1.16%.
  • Tea Leaf Disease Images: PSNR improved by 3.55% and SSIM by 2.48%.

Qualitatively, SM-CycleGAN produced images with fewer artifacts and more realistic color reproduction compared to Pix2Pix or CUT, which often struggled with background interference. Furthermore, the addition of self-attention improved training stability, smoothing out loss function oscillations in later stages without significantly increasing computational overhead.

Conclusion

SM-CycleGAN effectively leverages self-attention to overcome the limitations of local convolution in agricultural image translation. By capturing global context, the model restores color distortions and augments sparse datasets with high fidelity. Future work will focus on expanding the dataset to a wider variety of crops and improving performance on complex, noisy agricultural backgrounds.

Summary updated

Abstract

Crop disease detection and crop baking stage judgement require large image data to improve accuracy. However, the existing crop disease image datasets have high asymmetry, and the poor baking environment leads to image acquisition difficulties and colour distortion. Therefore, we explore the potential of the self-attention mechanism on crop image datasets and propose an innovative crop image data-enhancement method for recurrent generative adversarial networks (GANs) fused with the self-attention mechanism to significantly enhance the perception and information capture capabilities of recurrent GANs. By introducing the self-attention mechanism module, the cycle-consistent GAN (CycleGAN) is more adept at capturing the internal correlations and dependencies of image data, thus more effectively capturing the critical information among image data. Furthermore, we propose a new enhanced loss function for crop image data to optimise the model performance and meet specific task requirements. We further investigate crop image data enhancement in different contexts to validate the performance and stability of the model. The experimental results show that, the peak signal-to-noise ratio of the SM-CycleGAN for tobacco images and tea leaf disease images are improved by 2.13% and 3.55%, and the structural similarity index measure is improved by 1.16% and 2.48% compared to CycleGAN, respectively.
BibTeX
@inproceedings{liu2024sm,
  author = {Dian Liu and Yang Cao and Jing Yang and Jianyu Wei and Jili Zhang and Chenglin Rao and Banghong Wu and Dabin Zhang},
  title = {{SM-CycleGAN: crop image data enhancement method based on self-attention mechanism CycleGAN}},
  booktitle = {Scientific},
  year = {2024},
  doi = {10.1038/s41598-024-59918-3},
}
Figure