Efficient Dataflow Modeling of Peripheral Encoding

Rachel Brown, Vasha DuTell, Bruce Walter, Ruth Rosenholtz, Peter Shirley, Morgan McGuire, David LuebkeJournal of Vision 2022 · DOI

TL;DR

This paper introduces a novel, efficient dataflow model for simulating how the human visual system processes information in its periphery, particularly accounting for the phenomenon of "crowding." By optimizing computation and incorporating new visual features, their model significantly speeds up the creation of "metamer" images that are perceptually indistinguishable from originals in the periphery, paving the way for more realistic graphics and vision science research.

What Problem Does This Paper Address?

Our vision is remarkably adept at focusing on a small central area (the fovea) while processing the rest of our visual field in a much coarser manner. This peripheral vision is crucial for tasks like scene understanding and guiding our gaze, but it's also prone to "crowding" – where nearby stimuli make it difficult to discern a target. While graphics applications like foveated rendering can leverage this, existing models of peripheral vision and crowding are computationally expensive and rigid, hindering large-scale experiments and easy adaptation. This paper aims to bridge this gap by providing a faster, more flexible model that better captures the nuances of peripheral encoding, making it practical for computer graphics and vision science.

Methods

The core of this work is a new dataflow computational model for peripheral encoding. Unlike previous approaches that relied on complex pooling or contrast sensitivity functions, their model uses a PyTorch-based implementation that leverages automatic differentiation for efficient computation. Key methodological advancements include:

  • Efficient Dataflow Architecture: Mimicking deep learning pipelines, this allows for faster processing and easier modification of the model's components.
  • Augmented Steerable Pyramids: This technique decomposes images into components based on scale and orientation, capturing edge-like features.
  • Inclusion of "End-Stopped" Features: Previous models often missed these features, which are crucial for detecting where edges end or curve. The authors introduce new statistics to capture this.
  • Gaze-Centric Pooling: Pooling regions (where statistics are averaged) dynamically adjust in size based on the viewer's gaze point, mimicking how peripheral vision expands with eccentricity. This is achieved efficiently by warping the image into a log-polar space.

Datasets

  • Rendered Texture Dataset: The authors curated a dataset of 400 physically-based rendering (PBR) textures across 20 material categories. These were rendered at high resolution (1920x1080) under various lighting conditions to provide a diverse and realistic set for testing.
  • Photographic Cloud Data: 20 high-resolution photographs of clouds were also included.

Key Insights and Ideas

  • Peripheral Vision is More Than Just Low Resolution: The paper strongly emphasizes that crowding and other peripheral vision phenomena are not simply due to a lack of detail but involve complex encoding mechanisms.
  • "Crowding" is a Dominant Factor: The research highlights crowding as a key limitation in peripheral vision, driving the need for models that explicitly account for it.
  • Dataflow Models Enable Scalability: The shift to a dataflow architecture is a significant enabler, drastically reducing computation time and allowing for much larger-scale experiments than previously feasible.
  • End-Stopping Matters: The inclusion of statistics for "end-stopped" features, which capture edge terminations and curves, demonstrably improves the model's ability to generate perceptually accurate metamers.
  • Texture Properties Predict Metamer Success: The psychophysical evaluation revealed that certain texture properties, like high contrast and high roughness, lead to more successful (harder to detect) metamers, suggesting specific visual cues are more robustly encoded.

Key Results

  • Computational Speedup: The new dataflow model can synthesize 400 full-color 1920x1080 images in approximately 23 hours on a single GPU, a massive improvement over previous methods that could take hours for a single image.
  • Psychophysical Validation: A study with 29 participants showed an average accuracy of 64% in distinguishing metamer textures from originals. While this indicates the model isn't perfect, it provides a strong basis for evaluation.
  • Texture Descriptor Insights: The study identified significant interactions between texture properties and metamer success, particularly between contrast/roughness and coarseness/roughness. Textures with high contrast and roughness were generally more successful metamers.
  • Material Category Performance: While material categories themselves didn't strongly predict metamer success, snow/smoke and painted textures were generally harder to match, while plastic and tiled textures were easier.

Limitations and Future Work

  • Static Model: The current model is primarily for static images. The authors acknowledge that extending it to account for temporal dynamics (how vision changes over time) is a crucial next step, as crowding also occurs in time.
  • Fixation Accuracy: Remote psychophysical experiments inherently lack the precise control of eye-tracking, meaning participants' fixation accuracy could not be perfectly validated.
  • Texture Synthesis Challenges: The authors note that some textures, particularly those with high cross-scale regularity, remain challenging to synthesize accurately. They are exploring better statistics and synthesis approaches.
  • Call to Action: The paper serves as a call for further research, emphasizing the need for more robust and temporal models, and suggesting that their efficient framework can facilitate this progress. They plan to release their code to encourage further development.

Summary updated

Abstract

Identifying a visual stimulus in the periphery can be substantially impaired by the presence of other, nearby stimuli -- a phenomenon called “crowding”. Existing crowding models require significant compute time and their structure does not support easy modification, making it difficult to conduct large scale perceptual validation experiments. We advance crowding models of peripheral vision using a more efficient dataflow computational strategy which also simplifies adding and removing features. We leverage this flexibility by adding Gaussian filters that reduce ringing and encoding “end stopped” features which were missing from previous models. We evaluate our model in the context of texture recognition and letter crowding tasks in the periphery. Our improved computational framework may simplify development and testing of more sophisticated, complete models in more robust and realistic settings relevant to human vision and computer graphics.
BibTeX
@article{brown2022efficient,
  author = {Rachel Brown and Vasha DuTell and Bruce Walter and Ruth Rosenholtz and Peter Shirley and Morgan McGuire and David Luebke},
  title = {{Efficient Dataflow Modeling of Peripheral Encoding}},
  journal = {Journal of Vision},
  year = {2022},
  doi = {10.1167/jov.22.14.4378},
}
Figure