Classical information theory, pioneered by Claude Shannon, treats communication as the problem of reproducing at one point either exactly or approximately a message selected at another point. This "Level-A" communication focuses on bit-perfect reconstruction. However, as Kai Niu 0001 and Ping Zhang argue in Towards Semantic Communications: A Paradigm Shift, this approach ignores the "Level-B" problem: the transmission of meaning.
By shifting the goal from bit-level accuracy to "meaning-perfect" reconstruction, researchers are developing a new framework called Semantic Information Theory (SIT). This framework allows for compression ratios that significantly exceed traditional Shannon limits by leveraging the inherent redundancy of synonymous expressions.
The Intuition: Semantic Entropy is Lower than Shannon Entropy
In classical coding, every unique symbol or bit pattern is treated as a distinct entity that must be uniquely identified. If you have two words, "automobile" and "car," a standard Huffman coder treats them as two different events with their own probabilities.
However, if the goal of the communication is simply to convey the concept of a vehicle, these two words are functionally identical. Jin Xu and Zijian Liang demonstrate in Semantic Huffman Coding using Synonymous Mapping that we can group these "synonymous" symbols into a single set.
When we map multiple syntactic symbols to a single semantic representation, the "uncertainty" of the source decreases. Mathematically, this means semantic entropy is consistently lower than or equal to classical Shannon entropy. By calculating entropy based on the frequency of meanings rather than individual tokens, we lower the theoretical floor for how many bits are required to represent a message.
The Mechanism: Many-to-One Synonymous Mapping
The core mechanism of this research direction is the synonymous mapping function. This function acts as a many-to-one transformation that collapses the high-dimensional space of raw data (syntax) into a more compact space of intent (semantics).
- Grouping Synonyms: Distinct sequences that convey the same underlying intent are grouped into "synonymous sets."
- State Aggregation: In Semantic Markov Chain Using Synonymous Mapping, Jin Xu and colleagues extend this to sequences. Instead of tracking transitions between raw symbols, the model tracks transitions between semantic states, further reducing sequence uncertainty.
- Semantic Coding: Once the mapping is established, traditional coding algorithms can be "semanticized." For example, Semantic Arithmetic Coding (SAC) uses these synonymous mappings to achieve higher efficiency than standard arithmetic coding by ignoring variations that do not change the interpretation.
Beyond Text: Knowledge Graphs and Model Training
This principle of semantic redundancy isn't limited to natural language. It can be applied to any structured data where the same "fact" might be represented multiple times.
- Knowledge Graphs: Damien Graux and Diego Collarana applied these concepts to Knowledge Graph versions. By using Formal Concept Analysis to identify redundant facts across different versions of a graph, they achieved an 80% reduction in data size while keeping the graph fully queryable.
- Language Modeling: In Proxy Compression for Language Modeling, Lin Zheng and Lingpeng Kong use "invariant semantic compression" during training. By forcing a model to align raw byte-level inputs with compressed "proxy" views, the model learns to ignore low-level formatting noise and focus on high-level semantic features.
Reliable Communication in Extreme Conditions
The ultimate utility of this shift is formalized in the Semantic Joint Source Channel Coding (SJSCC) theorem. In Semantic Joint Source Channel Coding, Kai Niu proves that by focusing on meaning, communication systems can operate in environments with extremely low signal-to-noise ratios where traditional bit-level communication would fail.
Because the system only needs to transmit the "core intent" rather than every specific bit, it becomes inherently more robust. If the receiver can reconstruct a "synonym" of the original message that carries the exact same meaning, the communication is considered successful, even if the bits don't match. This paradigm shift effectively expands the achievable code rate, paving the way for the next generation of "intellicise" (intelligent and concise) communication networks.