Characterizing the Large-Scale Structure of Grounded Semantic Networks

Raja Marjieh, Pol van Rijn, Ilia Sucholutsky, Harin Lee, Nori Jacoby, Thomas L. Griffiths

Abstract

Humans organize semantic knowledge into complex networks that encode relations between concepts. The structure of those networks has broad implications for human cognitive processes and theories of semantic development. While lexical networks derived from word associations suggest a "small-world" architecture, the extent to which this reflects environmental regularities is less understood. We analyzed semantic descriptors from four large datasets of naturalistic stimuli (visual, auditory, and audio-visual) to construct "grounded" semantic networks. Our results show these networks exhibit small-world properties with a degree distribution best captured by a truncated power law. These networks are predictive of human sensory judgments and reaction times, suggesting that the structure of semantic networks is grounded in the statistical regularities of the environment.

The Scale-Free Debate and Environmental Grounding

For decades, cognitive scientists have modeled human knowledge as a semantic network where concepts (nodes) are connected by relationships (edges). Traditionally, these networks are built using "lexical" data, such as word association tasks. While these lexical networks consistently show "small-world" properties—characterized by high clustering but short paths between any two points—two major questions remained: the validity of the "scale-free" property and the role of environmental grounding.

Comparison of network topologies: Hierarchical, Random, Small-World, and Scale-Free.

The "scale-free" property implies that the number of connections (degree) follows a power-law distribution, where a few "hubs" have massive connectivity. However, whether these networks are truly scale-free or if this is a statistical artifact remains debated. Furthermore, it was unclear to what extent these structures reflect the way the world actually looks and sounds, rather than just abstract linguistic usage.

Methods: Constructing Grounded Networks

The researchers constructed networks directly from human descriptions of naturalistic sensory stimuli across multiple modalities. They employed two primary methods to elicit semantic descriptors:

  1. STEP-Tag (Sequential Transmission Evaluation Pipeline): A collaborative "algorithm with people" where one participant proposes tags for a stimulus, and subsequent participants rate their relevance or add new ones. This iterative process filters noise and converges on high-quality consensus descriptors.
  2. Free Captions: As a control, participants provided open-ended descriptions.
The STEP-Tag pipeline: an iterative process where participants propose, rate, and refine descriptors for a stimulus.

Network Construction: Undirected, unweighted graphs were built where each unique word is a node. An edge is created between two words if they co-occur as descriptors for the same stimulus. To ensure statistical rigor, the study used maximum likelihood estimation and log-likelihood ratio tests to compare different distributional models (e.g., Power Law vs. Truncated Power Law).

Datasets

The study utilized four large-scale naturalistic datasets comprising 7,916 stimuli and over 610,000 human responses:

  • BOLD5000: ~5,000 images of diverse indoor and outdoor scenes.
  • Mini-Kinetics: 1,000 short video clips of everyday human activities.
  • WikiArt: 1,000 modern and abstract paintings.
  • Emotional Prosody (RAVDESS): 1,000 audio recordings of actors speaking with varying emotional tones.

Key Insights and Results

Small-World Topology

Multimodal networks exhibit the same small-world topology as lexical networks, with high clustering coefficients (>0.7) and short average path lengths (<3 steps). This suggests that the "search efficiency" of our mental maps is an adaptation to how features co-occur in nature.

The Truncated Power Law

The degree distribution is best captured by a truncated power law. While "hubs" exist (e.g., man, water, white), they are less hyper-connected than a pure power law predicts. This truncation likely stems from physical or cognitive constraints—there is a limit to how many distinct contexts a single concept can inhabit. In the BOLD5K and Emotional Prosody datasets, the truncated power law significantly outperformed all other candidate distributions ($p < 0.0001$).

Behavioral and Predictive Validity

These networks predict human behavior. A word's "degree" in these multimodal networks significantly predicts how fast a human can identify that word in a lexical decision task (Spearman $\rho$ between -0.15 and -0.42). Essentially, the more "connected" a word is in our sensory experience, the faster our brains process it. Furthermore, text embeddings derived from these tags predicted human similarity judgments with high accuracy.

Conclusion and Future Work

Our findings shed new light on the origins of semantic network structure by grounding it in environmental regularities. While the datasets are large, they remain thematically bounded. Future work should scale toward a "visual WordNet" and investigate cross-cultural regularities. Additionally, linking these semantic "hubs" to neural data (e.g., fMRI) could reveal how these structures are physically represented in the brain.

Summary updated

Abstract

Humans organize semantic knowledge into complex networks that encode relations between concepts. The structure of those networks has broad implications for human cognitive processes, and for theories of semantic development. Evidence from large lexical networks such as those derived from word associations suggest that semantic networks are characterized by high sparsity and clustering while maintaining short average paths between concepts, a phenomenon known as a `small-world' network. It has also been argued that those networks are 'scale-free', meaning that the number of connections (or degree) between concepts follows a power-law distribution whereby most concepts have few connections while a few have many. However, the scale-free property is still debated, and the extent to which the lexical evidence reflects the naturally occurring semantic regularities of the environment has not been investigated systematically. To address this, we collected and analyzed semantic descriptors, human evaluations, and similarity judgments from four large datasets of naturalistic stimuli across three modalities (visual, auditory, and audio-visual) comprising 7,916 stimuli and 610,841 human responses. By connecting concepts that co-occur as descriptors of the same stimuli, we construct `grounded' semantic networks. We show that these networks exhibit a clear small-world structure with a degree distribution that is best captured by a truncated power law (i.e., the most-connected concepts are less common than predicted by a perfect power law). We further show that these networks are predictive of human sensory judgments on these domains, as well as reaction times in an independent lexical decision task. Finally, we show that grounded networks also share overlapping themes with previously analyzed lexical networks, which upon a more rigorous re-analysis are revealed to be truncated too. Our findings shed new light on the origins of the structure of semantic networks by grounding it in the semantic regularities of the environment.
BibTeX
@misc{marjieh2024characterizing,
  author = {Raja Marjieh and Pol van Rijn and Ilia Sucholutsky and Harin Lee and Nori Jacoby and Thomas L. Griffiths},
  title = {{Characterizing the Large-Scale Structure of Grounded Semantic Networks}},
  year = {2024},
  doi = {10.31234/osf.io/phte5},
  howpublished = {PsyArXiv},
}
Figure