Modality Invariant Conceptual Representations

91 concepts57 papers

When you read the word "ocean," see a photograph of crashing waves, or hear the sound of the surf, your brain does not treat these inputs as three unrelated experiences. Instead, your conscious awareness immediately converges on a single, unified concept. How do biological and artificial systems discard raw sensory formats—pixels, audio frequencies, or text tokens—to extract a shared meaning?

The answer lies in modality-invariant representation: the process by which a system maps diverse sensory streams into a centralized, abstract conceptual space. By understanding how the brain and deep neural networks strip away sensory-specific details, we can understand why cross-modal transfer, zero-shot learning, and universal semantic decoding are possible.

The Architecture of Convergence

To understand why modality invariance works, we must look at how information is routed. In both human brains and deep artificial networks, processing is hierarchical. Early stages are highly specialized: the primary visual cortex processes edges and orientations, while early layers of an audio network process raw frequencies.

However, as signals travel deeper into the network, they undergo a process of hierarchical aggregation. Patrick Taylor demonstrated in The global landscape of cognition: hierarchical aggregation as an organizational principle of human cortical networks and functions that as cognitive signals move away from sensory boundaries and deeper into the brain's structural network, they shift from concrete sensations to modality-independent symbolic meanings.

This aggregation terminates in specialized "amodal hubs"—high-level cortical regions that process meaning regardless of the input channel. For instance, Fatma Deniz et al. showed in The Representation of Semantic Information Across Human Cerebral Cortex During Listening Versus Reading Is Invariant to Stimulus Modality that fine-grained semantic maps in the human cortex are nearly identical whether a narrative is read or heard.

[Visual Input: Pixels]  --> [Early Visual Layers]  ---\
                                                       --> [Amodal Hub / Shared Latent Space]
[Auditory Input: Audio] --> [Early Auditory Layers] --/

This convergence is driven by specific subnetworks. While early sensory networks distinguish between watching a movie and listening to a story, the Default Mode Network (DMN) acts as a central hub to integrate the core semantic meaning of the narrative, as shown by Aurora Rossi et al. in Understanding the Importance of Brain Subnetworks with Shapley Values During Narrative Processing.

The Emergence of Invariant Neurons

What does this look like at the level of individual units? When a system is forced to categorize the world across multiple senses, it spontaneously develops concept-specific units that fire in response to an abstract idea, completely detached from its presentation format.

In neuroscience, these are often called "Halle Berry neurons." The authors of How cortical neurons help us see: visual recognition in the human brain observed that high-level neurons in the medial temporal lobe exhibit extreme invariance, activating for a specific concept whether it is presented as a photograph, a line drawing, or a written name.

This exact phenomenon emerges in artificial neural networks when they are trained on multimodal objectives. The authors of Deep Sparse Coding for Invariant Multimodal Halle Berry Neurons demonstrated that hierarchical sparse coding with top-down feedback naturally yields artificial units that achieve high levels of cross-modal invariance.

Similarly, when models are trained with joint language and vision supervision, they develop specialized neurons that respond to abstract semantic concepts (such as "music" or "morbidity") rather than specific visual geometries, as noted by the authors of What do Vision Transformers Learn? A Visual Exploration. This structural capacity is so fundamental that even text-only transformers contain latent visual-concept neurons that can be unlocked when paired with a simple vision encoder without altering the core weights of the model, as discovered by Sarah Schwettmann et al. in Multimodal Neurons in Pretrained Text-Only Transformers.

Cross-Modal Transfer and Decoding

Because high-level representations align into a shared mathematical space, knowledge gained from one sensory modality can be seamlessly transferred to another. This explains why we can build modality-agnostic decoders and perform zero-shot classifier transfer.

In computational neuroscience, this alignment allows predictive models to cross sensory boundaries. Jerry Tang et al. demonstrated in Brain encoding models based on multimodal transformers can transfer across language and vision that an encoding model trained strictly on brain responses to spoken stories can successfully predict how the brain will react to watching visual movies.

[Train Decoder] --> Brain Activity from Spoken Stories ---> Map to Semantic Space
                                                                   |
                                                                   v
[Test Decoder]  --> Brain Activity from Silent Movies  ---> Correctly Decodes Concept

On the decoding side, Mitja Nikolaus et al. showed in Modality-Agnostic fMRI Decoding of Vision and Language that a single decoder can interpret brain activity to identify a concept without knowing in advance whether the subject saw an image or read a text description.

This cross-modal transfer works because multimodal artificial networks (like CLIP) share the same representational geometry as the human brain's amodal hubs. Milad Mozafari et al. found in Multimodal neural networks better explain multivoxel patterns in the hippocampus that models trained on both visual and linguistic data align significantly better with human hippocampal activity than models restricted to a single modality. By learning to associate diverse sensory inputs with unified semantic labels, these networks replicate the brain's abstract encoding mechanisms.

Abstract Schemas and Event Structures

Modality invariance is not limited to static objects or single words; it extends to complex, dynamic sequences and real-world "scripts." When we experience a narrative, our brains do not just record the sights or sounds sequentially; they construct an abstract situation model.

Christopher Baldassano et al. discovered in Discovering event structure in continuous narrative perception and memory that high-level cortical regions segment continuous experiences into discrete events identically, whether an individual is watching a visual movie or listening to an audio description. These high-level event schemas focus entirely on abstract meaning. As Uri Hasson et al. observed in Representation of Real-World Event Schemas during Narrative Perception, the neural patterns that represent a script (like dining at a restaurant) remain invariant across different stories, different people, and different media.

This cognitive principle has directly influenced machine learning architectures. To create universal representations across data types, the authors of MMEKG: Multi-modal Event Knowledge Graph towards Universal Representation across Modalities utilized events as a modality-agnostic framework. Because the core conceptual structure of an event remains stable whether it is depicted in an image or narrated in text, event-centric abstractions serve as a universal bridge for machine understanding.

Ultimately, whether looking at a single neuron activating for a name, an fMRI decoder translating thoughts across text and video, or a transformer aligning visual tokens into a language space, the underlying mechanism is the same: the extraction of semantic invariants from a multi-sensory world.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 57 papers
  2. Version 1 · 2026-10-04 · 69 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_modality_independent_representation_f07aa40f,
  title = {modality-independent representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/modality-independent-representation/rdr_f07aa40fd5604492b8e65efcc1643bed},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_modality_independent_representation_f07aa40f_v2,
  title = {modality-independent representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/modality-independent-representation/rdr_f07aa40fd5604492b8e65efcc1643bed/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}

See something wrong? Report a mistake