Unimodal statistical learning produces multimodal object-like representations

Gábor Lengyel, Goda M. Žalalytė, Alexandros Pantelides, James N. Ingram, József Fiser, Máté Lengyel, Daniel M. WolperteLife 2019 · DOI

TL;DR

This study demonstrates that humans can learn to perceive "objects" based purely on statistical regularities in visual or haptic sensory input, even without explicit boundary cues. This learning leads to flexible, cross-modal representations, allowing participants to predict haptic properties from visual statistics and vice-versa, suggesting a fundamental mechanism for object perception.

What Problem Does This Paper Address?

How do we form abstract "object" representations from raw sensory data? Traditional views emphasize visual or haptic boundaries as crucial for segmenting the world into discrete objects. However, these cues can be ambiguous (e.g., zebra stripes) or insufficient (e.g., illusory contours). This paper challenges the necessity of explicit boundaries, proposing that consistent statistical properties across sensory inputs are a more fundamental principle. The authors investigate whether learning these statistical regularities, in the absence of clear boundaries, can lead to representations that generalize across sensory modalities, akin to how we perceive real-world objects.

Methods

The researchers developed a novel "visuo-haptic statistical learning paradigm." Participants were exposed to artificial "objects" defined by consistent co-occurrence statistics of shapes, either through visual observation or physical interaction (haptic pulling). Crucially, these artificial objects lacked explicit visual or haptic boundaries that would normally signal objecthood.

Two main experiments were conducted:

  1. Visual Statistical Exposure: Participants viewed sequences of scenes composed of abstract shapes. They learned the statistical relationships between shapes that formed "true pairs" (objects) solely through visual input.
  2. Haptic Statistical Exposure: Participants interacted with scenes using a robotic interface (vBOT), learning statistical relationships through the forces required to pull apart shape configurations.

After exposure, participants were tested on two tasks:

  • Visual Familiarity Test: Participants judged which of two presented pairs was more familiar, indicating their learned visual representations.
  • Haptic Pulling Test: Participants exerted force to pull apart shape configurations, allowing researchers to measure how well they predicted the required "breakage force" based on learned haptic statistics.

A key aspect was testing "zero-shot" cross-modal generalization: predicting haptic properties from visual learning, and vice-versa, without any direct training or feedback in the generalization task.

Datasets

The study utilized custom-generated datasets of artificial visual stimuli (abstract shapes arranged in pairs and scenes). No pre-existing datasets were used for training or evaluation. The "inventory" consisted of abstract geometric shapes, which were combined into "true pairs" (representing objects) and "pseudo pairs" (control stimuli). The specific configurations of these pairs formed the "scenes" used in the exposure and testing phases.

Key Insights and Ideas

  • Boundaries are not essential: The most striking finding is that explicit visual or haptic boundaries are not necessary for humans to segment scenes into object-like representations. Statistical regularities alone are sufficient.
  • Statistical learning creates "object-like" representations: The learned representations are not merely abstract "units" but exhibit properties akin to real objects, demonstrated by their ability to generalize across senses.
  • "Zero-shot" cross-modal generalization: Participants could predict haptic properties (force required to pull apart) from visual statistical learning, and vice-versa, without any direct training in the tested modality. This suggests a unified, modality-independent representation.
  • Implicit learning is key: The generalization effect persisted even when controlling for participants' explicit knowledge of the learned pairs, highlighting the role of implicit statistical learning.
  • Consistency drives generalization: Individual differences in performance on both visual and haptic tasks were highly correlated, suggesting that the same underlying learning mechanism drives performance across modalities.

Key Results

  • Within-modality learning: Participants showed strong learning in both visual familiarity tests (average performance ~0.77 after visual exposure) and haptic pulling tests (average correlation ~0.27 after visual exposure, ~0.28 after haptic exposure).
  • Cross-modal generalization:
    • Visual learning significantly predicted haptic pulling performance (correlation between visual familiarity and haptic pulling performance was high, with a rectified exponential-binomial fit explaining 87% of the visual familiarity range).
  • Haptic learning also generalized to visual familiarity judgments, though this effect was weaker when visual and haptic statistics were in conflict.
  • High correlation between tasks: The performance on the visual familiarity test and the haptic pulling test was highly correlated across participants (R = 0.84).
  • Implicit transfer dominates: A significant partial correlation between visual and haptic performance remained even after controlling for explicit knowledge, indicating that implicit transfer accounted for a larger portion (67%) of the generalization effect.

Limitations and Future Work

  • Artificial stimuli: The study used artificial shapes and stimuli, which may not fully capture the complexity of real-world object perception. Future work could explore these principles with more naturalistic stimuli.
  • "Objectness" continuum: The paper acknowledges that "objectness" exists on a continuum. Future research could investigate how statistical learning contributes to segmenting entities with varying degrees of objectness.
  • Mechanism of generalization: While the study demonstrates the effect, it doesn't pinpoint the exact neural or computational mechanism underlying the modality-general representation. Further investigation into the neural basis of this cross-modal integration is warranted.
  • Infant development: The findings in adults suggest that similar mechanisms might operate in infants, offering a potential avenue for developmental research on how object representations emerge.

Summary updated

Abstract

The concept of objects is fundamental to cognition and is defined by a consistent set of sensory properties and physical affordances. Although it is unknown how the abstract concept of an object emerges, most accounts assume that visual or haptic boundaries are crucial in this process. Here, we tested an alternative hypothesis that boundaries are not essential but simply reflect a more fundamental principle: consistent visual or haptic statistical properties. Using a novel visuo-haptic statistical learning paradigm, we familiarised participants with objects defined solely by across-scene statistics provided either visually or through physical interactions. We then tested them on both a visual familiarity and a haptic pulling task, thus measuring both within-modality learning and across-modality generalisation. Participants showed strong within-modality learning and 'zero-shot' across-modality generalisation which were highly correlated. Our results demonstrate that humans can segment scenes into objects, without any explicit boundary cues, using purely statistical information.
BibTeX
@inproceedings{lengyel2019unimodal,
  author = {Gábor Lengyel and Goda M. Žalalytė and Alexandros Pantelides and James N. Ingram and József Fiser and Máté Lengyel and Daniel M. Wolpert},
  title = {{Unimodal statistical learning produces multimodal object-like representations}},
  booktitle = {eLife},
  year = {2019},
  doi = {10.7554/elife.43942},
}

Loading related content...

Figure