Synthesis and perceptual scaling of high-resolution naturalistic images using Stable Diffusion

Leonardo Pettini, Carsten Bogler, Christian F. Doeller, John­–Dylan Haynes

TL;DR

This paper introduces a novel method for generating high-resolution, naturalistic images that vary along a continuous perceptual scale using Stable Diffusion XL. This approach, validated by human perception and working memory tasks, offers a flexible and powerful tool for studying visual perception, attention, and memory in ecologically relevant settings.

What Problem Does This Paper Address?

Understanding how humans perceive and remember naturalistic scenes is crucial for ecological validity in cognitive research. However, creating controlled stimulus sets that capture the richness of real-world images while allowing for systematic manipulation of perceptual and semantic properties has been a significant challenge. Traditional methods often rely on discrete images or low-level feature morphing, which can lead to semantic ambiguity or lack fine-grained perceptual control. While GANs have offered improvements, this paper addresses the need for a more flexible and high-resolution approach to generate stimuli that vary smoothly along a perceptually scaled continuum.

Methods

The authors leverage Stable Diffusion XL (SDXL), a powerful text-to-image diffusion model, to synthesize photorealistic "object-scenes." The core of their method involves:

  1. Text-to-Image Generation: Using carefully crafted text prompts to generate diverse images of objects within coherent scenes, ensuring a clear central object and no human figures.
  2. Perceptual Similarity Estimation (LPIPS): Employing the Learned Perceptual Image Patch Similarity (LPIPS) metric to quantify the perceptual differences between generated images.
  3. Noise Latent Interpolation: Generating a continuum of images by interpolating between the "noise latents" of two selected anchor images. This method preserves semantic meaning while varying perceptual features.
  4. Human Validation: Conducting a large-scale online "method of triads" similarity judgment experiment to validate and fine-tune the LPIPS-based perceptual ordering.
  5. Working Memory Task: Using the validated stimulus set in a delayed match-to-sample task to assess how perceptual similarity influences stimulus confusability and working memory performance.

Datasets

  • Newly Introduced Stimulus Set: The paper introduces a novel dataset of 1080 images, comprising 10 variants for each of 108 object scenes across six categories (animals, plants, landscape elements, items, buildings, vehicles). These images are ordered along a validated perceptual continuum.
  • No external datasets were used for training the diffusion model itself, as SDXL is a pre-trained model.

Key Insights and Ideas

  • Diffusion Models as a Flexible Tool: Stable Diffusion XL proves to be a highly effective tool for generating high-resolution, semantically consistent images with fine-grained perceptual variations, offering greater flexibility than previous methods like GANs or manual image selection.
  • Noise Latent Interpolation for Perceptual Control: Interpolating in the noise latent space, rather than the text prompt space, is key to maintaining semantic consistency while smoothly varying perceptual features.
  • LPIPS as a Strong Initial Predictor: The LPIPS metric serves as a valuable and efficient first step for ordering images perceptually, aligning well with human judgments.
  • Human Validation is Crucial: While LPIPS is a good starting point, human psychophysical judgments are essential for fine-tuning the perceptual scale and ensuring ecological relevance.
  • Perceptual Similarity Predicts Cognitive Performance: The validated perceptual ordering of stimuli directly predicts performance in a working memory task, demonstrating the ecological validity of the generated stimulus set for studying cognitive processes.

Key Results

  • Successful Generation of Perceptual Continua: The authors successfully generated sets of 10 images for each of 108 object scenes, ordered along a perceptually continuous scale.
  • High Agreement Between LPIPS and Human Judgments: The LPIPS-based ordering showed strong alignment with human similarity judgments, with a Spearman rank correlation of ρ = 0.73.
  • Working Memory Performance Scales with Perceptual Similarity: In a visual working memory task, accuracy decreased systematically as perceptual similarity between targets and foils increased, confirming the graded nature of perceptual confusability.
  • Low Artefact Rate: After visual inspection, only 3.3% of generated images were excluded due to artefacts, with most objects having very few exclusions.
  • Category-Specific Artefact Tendencies: Animal and item categories showed higher exclusion rates, often due to artefacts like incorrect limb counts or non-realistic perspectives.

Limitations and Future Work

  • Subjectivity in Prompt Selection: While efforts were made to standardize prompts, the initial selection of objects and prompts could introduce some subjectivity. Future work could explore data-driven concept selection frameworks.
  • Artefact Variability: Artefacts, though generally low, varied across categories and objects, suggesting potential differences in training data availability or prompt engineering challenges.
  • Limited Control Over Specific Perceptual Dimensions: The current approach generates variations along a general perceptual continuum without explicit control over specific dimensions (e.g., only controlling lightness). Future work could explore extensions of diffusion models or GANs for more targeted control.
  • Generalizability to Complex Scenes: The object-centric design, while beneficial for controlled studies, limits applicability for research on complex, multi-object scene perception or naturalistic gaze behavior.
  • LPIPS Magnitude Interpretation: The magnitude of LPIPS scores across different object scenes was less interpretable, suggesting future work could address this by accounting for similarity scores both within and across object scenes.
  • Future Directions: The authors suggest extending the pipeline to generate scene variants, augmenting existing datasets, and exploring the cognitive implications of category-specific artefact differences. They also highlight the potential for using this method to study scene perception and feature binding.

Summary updated

Abstract

Naturalistic scenes are of key interest for visual perception, but controlling their perceptual and semantic properties is challenging. Previous work on naturalistic scenes has frequently focused on collections of discrete images with considerable physical differences between stimuli. However, it is often desirable to assess representations of naturalistic images that vary along a continuum. Traditionally, perceptually continuous variations of naturalistic stimuli have been obtained by morphing a source image into a target image. This produces transitions driven mainly by low-level physical features and can result in semantically ambiguous outcomes. More recently, generative adversarial networks (GANs) have been used to generate continuous perceptual variations within a stimulus category. Here, we extend and generalize this approach using a different machine learning approach, a text-to-image diffusion model (Stable Diffusion XL), to generate a freely customizable stimulus set of photorealistic images that are characterized by gradual transitions, with each image representing a unique exemplar within a prompted category. We demonstrate the approach by generating a set of 108 object scenes from six categories. For each object scene, we generate ten variants that are ordered along a perceptual continuum. This ordering was first estimated using a machine learning model of perceptual similarity (LPIPS) and then subsequently validated with a large online sample of human participants. In a subsequent experiment, we show that this ordering is also predictive of stimulus confusability in a working memory task. Our image set is suited for studies investigating the graded encoding of naturalistic stimuli in visual perception, attention, and memory.
BibTeX
@inproceedings{pettini2025synthesis,
  author = {Leonardo Pettini and Carsten Bogler and Christian F. Doeller and John­–Dylan Haynes},
  title = {{Synthesis and perceptual scaling of high-resolution naturalistic images using Stable Diffusion}},
  booktitle = {Behavior},
  year = {2025},
  doi = {10.3758/s13428-025-02889-8},
}
Figure