Characterizing the Large-Scale Structure of Multimodal Semantic Networks

Raja Marjieh, Pol van Rijn, Ilia Sucholutsky, Harin Lee, Nori Jacoby, Thomas L. GriffithsCogn. Sci 2025 · DOI

TL;DR

This paper introduces "multimodal semantic networks" derived from human descriptions of sensory stimuli, revealing they share "small-world" and "truncated power-law" structures with traditional lexical networks. These findings suggest that the organization of human semantic knowledge is deeply rooted in the regularities of the environment itself, not just internal cognitive processes.

What Problem Does This Paper Address?

For decades, cognitive scientists have studied how humans organize knowledge into complex "semantic networks." Previous research, primarily using word association tasks, suggested these networks have a "small-world" structure (highly clustered with short paths between concepts) and a "scale-free" distribution of connections (a few "hubs" with many connections, most with few). However, these findings were based on curated lexical datasets, leaving open the question of whether they reflect the structure of semantic knowledge as it naturally arises from interacting with the real world. This paper aims to bridge this gap by constructing and analyzing semantic networks directly from human descriptions of diverse, naturalistic sensory stimuli across different modalities.

Methods

The authors employed a novel semantic mining procedure called STEP-Tag (Sequential Transmission Evaluation Pipeline for semantic tag mining). In this method, participants collaboratively and iteratively tag stimuli (images, videos, audio) with descriptive words. Tags are then rated for relevance by subsequent participants, and poorly rated tags are pruned. This process aims to converge on a set of high-quality, agreed-upon descriptors for each stimulus.

To construct the semantic networks, the co-occurrence of these descriptors for the same stimulus was used to create edges between words (nodes). These networks were then subjected to rigorous statistical analyses, including:

  • Network Statistics: Calculating average shortest path length, clustering coefficient, and the small-worldness coefficient ($\sigma$) to assess small-world properties.
  • Degree Distribution Analysis: Using maximum likelihood estimation to fit various distributions (power-law, exponential, log-normal, stretched exponential, truncated power-law) to the degree distribution (number of connections per word) and comparing them using log-likelihood ratio tests.
  • Community Detection: Employing the Louvain algorithm to identify clusters or "communities" within the networks.
  • Behavioral Evaluations:
    • Similarity Judgments: Using text embeddings of descriptors to predict human similarity judgments of stimuli and comparing these predictions to human inter-rater reliability.
  • Lexical Decision Tasks: Correlating node degree with reaction times in a lexical decision task to assess behavioral predictability.
  • Comparison to Lexical Networks: Reconstructing and analyzing traditional free association and WordNet networks for comparison.

Datasets

The study utilized four diverse datasets:

  • BOLD5000: A collection of 4916 unique images of indoor/outdoor activities and natural scenes.
  • Mini-Kinetics: 1000 short video clips of everyday activities.
  • WikiArt: 1000 artworks from the WikiArt Emotions dataset, focusing on abstract art.
  • Emotional Prosody: 1000 audio recordings of semantically neutral sentences conveying different emotions.

These datasets were chosen to cover naturalistic and complex domains, multiple perceptual modalities (visual, auditory, audio-visual), and both natural and abstract stimuli.

Key Insights and Ideas

  • Environmental Regularities Shape Semantics: The core finding is that semantic networks derived directly from sensory experiences exhibit structural properties (small-world, truncated power-law) similar to those found in traditional lexical networks. This strongly suggests that the organization of human semantic knowledge is heavily influenced by the statistical regularities present in the environment.
  • Multimodal Networks Mirror Lexical Networks: The study demonstrates that constructing semantic networks from diverse sensory inputs (not just word associations) yields networks with comparable structural characteristics to those derived from purely textual data.
  • Truncated Power-Law is Key: While previous work debated "scale-free" properties, this study robustly shows that a truncated power-law distribution (where extremely high-degree nodes are less common than a perfect power-law would predict) best captures the degree distribution across all multimodal datasets. This implies constraints on the growth of highly connected concepts.
  • Domain-Specific Structures: The manifestation of these properties can differ across domains, influenced by factors like indoor-outdoor distinctions in natural scenes or valence-arousal dimensions in emotional prosody.
  • Core Concepts as Organizers: The prominence of basic color concepts in multiple datasets suggests they may act as central organizing hubs that span across different modalities.

Key Results

  • Small-World Topology: All four multimodal semantic networks exhibited a strong small-world topology, characterized by high clustering coefficients (>.7) and short average path lengths (<3.5 steps), with small-worldness coefficients ($\sigma$) significantly greater than 1 (ranging from 20.5 to 160.8).
  • Truncated Power-Law Degree Distribution: Log-likelihood ratio tests consistently favored the truncated power-law model over other distributions (including a pure power-law) for the degree distribution of all multimodal networks. The variance explained by the truncated power-law model was very high (>.95).
  • Behavioral Predictivity:
    • Textual descriptors derived from multimodal networks significantly predicted human similarity judgments across all domains, with correlations comparable to human inter-rater reliability in some cases.
  • Node degree in these networks was negatively correlated with reaction times in a lexical decision task, meaning words with more connections were identified faster. This effect persisted even after controlling for word frequency, particularly for BOLD5K and free association networks.
  • Overlap with Lexical Networks: Multimodal networks shared thematic themes with traditional free association networks, especially for natural scenes (BOLD5K), indicating a common underlying organizational principle.

Limitations and Future Work

  • Thematic Boundedness: While diverse, the datasets are still thematically bounded. Future work could explore the persistence of observed regularities with larger datasets and more general big data methodologies.
  • STEP-Tag vs. Free Captions: The study acknowledges that STEP-Tag (focused on succinct, agreed-upon descriptors) and free captions (allowing individual variation) yield slightly different network statistics. Future research could explore the implications of these differences.
  • Generalizability: The current participant cohort was restricted to the United States. Cross-cultural and individual-differences research is needed to assess the generalizability of these findings.
  • Neural Data Integration: The paper sets the stage for future work by enriching the BOLD5000 dataset with rich semantic labels, enabling investigations into how these semantic networks are reflected in neural data.
  • Growth Mechanisms: While the study highlights environmental regularities, future work could explore how these regularities shape semantic acquisition and growth, potentially integrating models like preferential acquisition.
  • Similarity Prediction: Future work could explore using the network structure itself, rather than just text embeddings, to construct similarity predictions, especially for domains like abstract art where text-based predictions showed a larger gap compared to image-based ones.

Summary updated

BibTeX
@article{marjieh2025characterizing,
  author = {Raja Marjieh and Pol van Rijn and Ilia Sucholutsky and Harin Lee and Nori Jacoby and Thomas L. Griffiths},
  title = {{Characterizing the Large-Scale Structure of Multimodal Semantic Networks}},
  journal = {Cogn. Sci.},
  year = {2025},
  doi = {10.1111/cogs.70131},
}
Figure