Abstract: Semantic communication is a promising direction for the future of communication systems. Synonyms act as crucial bridges between semantics and syntax. Inspired by this, this paper introduces a semantic Markov chain model based on state synonymous mapping. This paper demonstrates that the syntactic Markov chain can be transformed into a semantic Markov chain through synonymous mapping. The stationary distribution of the semantic Markov chain can be obtained by partly summing the stationary distribution of the syntactic Markov chain. Similarly, the transition probability of the semantic Markov chain can be obtained by performing a weighted summation of the transition probabilities of the syntactic Markov chain. Additionally, we theoretically prove that the sequence entropy and the entropy rate of the semantic Markov source are no more than those of the syntactic Markov source.
Introduction
Classical Information Theory (CIT), pioneered by Claude Shannon, has governed communication for over 70 years by focusing on the technical level: ensuring bits are transmitted accurately regardless of their meaning. As we approach the theoretical "Shannon limit" of bit-rate efficiency, researchers are looking toward semantic communication—the second of Weaver’s three levels of communication—to find new ways to optimize systems.
A major challenge in modeling real-world data like text is the "state-space explosion." If every unique word is a state in a Markov chain, the model becomes computationally unwieldy. Furthermore, many states are functionally identical for a given task (e.g., "shack" and "hut"). This paper addresses the lack of a unified theoretical framework to map these syntactic "synonyms" into a streamlined semantic model, presenting a formal mathematical bridge between syntax and semantics.
Methodology
The researchers propose a mathematical transformation to extend classical Markov chains into the semantic domain. This process involves three primary steps:
1. Synonymous Mapping
They define a one-to-many mapping where a single semantic state ($\tilde{u}$) represents a set of syntactic states ($U$). Crucially, "synonymy" here is task-dependent; it refers to any information that can be merged at the receiver's end without causing errors for the specific goal. This mapping bridges the gap between the semantic information space and the syntactic information space.
2. Semantic Markov Chain Construction
By grouping synonymous states, a syntactic Markov chain can be transformed into a semantic one. The authors derive formulas for the semantic stationary distribution (the sum of the stationary distributions of its constituent syntactic states) and the semantic transition probability matrix (a weighted summation of the original syntactic transitions).
3. Counter-fitting for Text
To validate the theory on real language, the authors use "counter-fitting" to adjust standard word vectors (like GloVe). This technique pulls synonyms closer in vector space while pushing antonyms apart. Cosine distance is then used as a threshold to determine which words belong in the same "synonymous set."
Theoretical Insights
- Entropy Reduction: The paper theoretically proves that the sequence entropy and entropy rate of a semantic Markov source are always less than or equal to the syntactic source. By focusing on meaning, there is less "uncertainty" to transmit, enabling better data compression.
- Model Consistency: A key finding is that if a syntactic Markov chain is homogeneous (stable over time), its semantic counterpart remains homogeneous. This ensures that the model's properties are preserved after grouping synonyms.
- The "Sebit": The authors introduce the term "sebits" (semantic bits) to distinguish semantic information measures from traditional Shannon bits.
- Task-Centricity: Synonymy is defined by the task. For instance, in a weather-monitoring system for solar power, "humidity" and "precipitation" might be synonymous if the only variable affecting the goal is "sunshine duration."
Experimental Results
The model was validated using a dataset derived from the book Harry Potter (84,418 words) and pre-trained GloVe word vectors.
- Compression Gains: 6,502 unique words were categorized into 4,898 synonymous sets using a cosine distance threshold of 0.2.
- Quantitative Drop in Entropy:
- Syntactic symbol entropy: 1.410 bits.
- Semantic symbol entropy: 0.7910 sebits.
- Entropy rate: Dropped from 1.0342 bits to 0.7842 sebits.
- Clustering Visualization: Using Principal Component Analysis (PCA), the authors demonstrated that as the cosine distance threshold increases, word vectors form distinct clusters, effectively compressing the language space while maintaining the Markov property.
Conclusion
This research provides a robust framework for semantic communication by formalizing the Semantic Markov Chain. While the current model primarily addresses word-level synonymy, it paves the way for future work in phrase-level analysis and context-aware semantic modeling. By reducing entropy through synonymous mapping, this approach offers a path toward more efficient, meaning-aware data transmission in next-generation networks.
Summary updated