Diptychs of human and machine perceptions

Vivien Cabannes, Thomas Kerdreux, Louis ThiryarXiv 2020 · arXiv

Abstract

This research introduces a creative framework for visualizing the "perception gap" between biological and artificial vision systems. By utilizing Voronoi-tessellated diptychs, the authors demonstrate how task-oriented AI focuses on discriminative features like silhouettes and edges, while human attention is captured by semantic context, faces, and artistic composition.

Introduction: Visualizing the Perception Gap

As deep learning models become increasingly integrated into creative and analytical fields, understanding the "black box" of machine perception is essential. While mathematical tools like saliency maps provide a view of what a neural network "attends" to, they are often too abstract for direct comparison with human experience. This paper addresses the lack of qualitative, intuitive comparisons between task-oriented AI (trained to categorize) and the contemplative human mind (driven by evolution and semantics) by turning these disparate modes of seeing into a shared visual language: artistic diptychs.

Methodology: Picture Tessellation

The authors developed a pipeline to transform raw images into "perception maps" using a technique called picture tessellation:

  1. Saliency Mapping: Machine saliency is calculated using the gradient of the network’s output with respect to the input image. Human saliency is collected via gaze focus data from eye trackers and existing fixation datasets.
  2. Point Sampling: Saliency maps are treated as probability density functions. High-interest areas receive a high density of sampled points, while "boring" areas receive few.
  3. Voronoi Tessellation: These points act as seeds for a Voronoi diagram. This creates a mosaic where high-attention areas are rendered with tiny, sharp tiles (high detail) and low-attention areas are rendered with large, blurry tiles (low detail).
  4. Coloring: Each tile is colored based on the RGB mean of the pixels it covers in the original image.

Comparative Insights: Human vs. Machine

The resulting diptychs reveal fundamental differences in how biological and artificial systems prioritize information. A primary example is the "flower" arrangement, where the two systems diverge significantly in their focus.

Human perception is consistently drawn to the semantic and aesthetic center of the image, such as the intricate textures of the flower petals.

In contrast, the machine's perception is driven by the need for classification. It often ignores the internal substance of an object in favor of the discriminative edges that define its category.

Machine perception focuses on the silhouette and high-contrast edges, such as the vase, which are critical for its classification task.

Key Findings

  • Silhouette vs. Substance: AI is hyper-focused on outlines for identification, often ignoring the internal colors and textures that humans find captivating.
  • Semantic Blindness: AI trained for classification frequently misses "narrative" details. For instance, in street scenes, the human eye is drawn to people, while the AI may ignore them if they do not contribute to its specific classification score (e.g., finding buildings or cars).
  • Compositional Awareness: Humans naturally follow vanishing points and perspective lines. AI perception is largely indifferent to these structural aesthetics unless they correlate with a learned feature.
  • The Adversarial Hint: In some cases, AI saliency is uniform across the frame, suggesting the classifier is sensitive to changes anywhere in the image—a property linked to the vulnerability of neural networks to adversarial attacks.

Artistic Utility and Future Directions

Beyond analysis, these perception maps serve as "computational catalysts." The authors demonstrated that by projecting these distorted maps onto a canvas, an artist can use the machine's unique "view" as a starting point for a new, hybrid painting.

While effective, the authors acknowledge that "saliency" remains a subjective and task-dependent metric. Future work could explore other rendering styles, such as pointillism or impressionism, to further refine how we visualize the divide between human and machine vision, ultimately working toward AI models that perceive the world in a more human-centric, semantic way.

Summary updated

Abstract

We propose visual creations that put differences in algorithms and humans \emph{perceptions} into perspective. We exploit saliency maps of neural networks and visual focus of humans to create diptychs that are reinterpretations of an original image according to both machine and human attentions. Using those diptychs as a qualitative evaluation of perception, we discuss some crucial issues of current \textit{task-oriented} artificial intelligence.
BibTeX
@misc{cabannes2020diptychs,
  author = {Vivien Cabannes and Thomas Kerdreux and Louis Thiry},
  title = {{Diptychs of human and machine perceptions}},
  year = {2020},
  eprint = {2010.13864},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2010.13864},
}
Figure