Abstract
The notion of salience has been singled out as the explanatory factor for a diverse range of linguistic phenomena. In particular, perceptual salience (e.g., visual salience of objects in the world, acoustic prominence of linguistic sounds) and semantic-pragmatic salience (e.g., prominence of recently mentioned or topical referents) have been shown to influence language comprehension and production. A different line of research has sought to account for behavioral correlates of cognitive load during comprehension as well as for certain patterns in language usage using information-theoretic notions, such as surprisal. Surprisal and salience both affect language processing at different levels, but the relationship between the two has not been adequately elucidated, and the question of whether salience can be reduced to surprisal/predictability is still open. Our review identifies two main challenges in addressing this question: terminological inconsistency and lack of integration between high and low levels of representations in salience-based accounts and surprisal-based accounts. We capitalize upon work in visual cognition in order to orient ourselves in surveying the different facets of the notion of salience in linguistics and their relation with models of surprisal. We find that work on salience highlights aspects of linguistic communication that models of surprisal tend to overlook, namely the role of attention and relevance to current goals, and we argue that the Predictive Coding framework provides a unified view which can account for the role played by attention and predictability at different levels of processing and which can clarify the interplay between low and high levels of processes and between predictability-driven expectation and attention-driven focus.
Introduction: The Predictability vs. Prominence Debate
In the study of psycholinguistics, two major concepts frequently compete to explain how humans process language: Surprisal and Salience.
Surprisal, rooted in Information Theory, posits that the cognitive load of a word is directly proportional to its unpredictability. Essentially, the more "surprising" a word is within its context, the harder the brain must work to process it. Salience, conversely, is a more heterogeneous term used to describe elements that "stand out" from their background. This can manifest as perceptual salience (such as a sudden loud noise or a bright color) or discourse salience (such as a recently mentioned topic or a referent that is central to the conversation).
While both concepts are influential, they often fail to align. A central challenge in current research is determining whether salience can be reduced to a form of predictability, or if it represents a distinct cognitive mechanism. This paper argues that current information-theoretic models are incomplete because they often ignore the critical roles of human goals and attention.
The Limitations of Pure Surprisal: The Snow-Screen Paradox
A primary critique of pure surprisal models is what we might call the "Snow-Screen Paradox." In a purely information-theoretic sense, a screen filled with random static (snow) is high-surprisal because every pixel is unpredictable. However, humans find such stimuli incredibly boring and quickly tune them out.
This suggests that the brain does not care about randomness for its own sake. Instead, it prioritizes Bayesian Surprise—not just how rare an event is, but how much that event forces us to update our internal model of the world. This leads to the "Goldilocks Effect," where humans and infants tend to ignore stimuli that are either too simple (low surprisal) or too noisy/complex (extremely high surprisal), focusing instead on the "just right" middle ground that provides the most useful information for learning.
A Unified Framework: Predictive Coding
To bridge the gap between salience and surprisal, we propose the Predictive Coding framework. This model views the brain as a hierarchy where top-down goals "weight" the importance of bottom-up signals.
In this hierarchical structure, higher levels (such as a speaker's perceived intent or the overarching goal of a conversation) send predictions down to lower levels (such as syntax or phonology). Lower levels only pass "prediction errors" back up the chain if those errors are deemed relevant to the higher-level goals. This creates a system of Neural Gating, likely facilitated by phase-amplitude coupling, where low-frequency brain waves (representing top-down attention) act as a gate that amplifies or suppresses high-frequency waves carrying bottom-up sensory information.
Key Insights and Theoretical Synthesis
Our review of visual cognition and neuroscience leads to several key conclusions for the future of language modeling:
- Salience as Attention: Salience in linguistics is essentially a description of the current focus of attention. This can be driven by bottom-up "pop-out" effects (perceptual salience) or top-down, goal-directed searches (relevance).
- Multilevel Surprisal: Surprisal should not be calculated as a single value for a word. Rather, it must be assessed at every level of the cognitive hierarchy—from acoustic and lexical to syntactic and situational.
- Relevance over Probability: The brain does not simply minimize prediction error; it minimizes weighted prediction error. The weights are determined by the relevance of the information to the task at hand.
Conclusion and Future Directions
While the Predictive Coding framework offers a theoretically sound path forward, significant work remains. Current computational models of language have yet to fully implement this hierarchical, goal-weighted approach. Furthermore, the role of affect—how rewarding or threatening a word is—remains a massive driver of attention that surprisal models currently ignore. Future research must focus on building "hierarchical surprisal models" that can distinguish between the cognitive load of a surprising sound and the cognitive load of a surprising conceptual shift, ultimately moving toward a more holistic understanding of how we navigate the complexities of human communication.
Summary updated