Disentangling Learnable and Memorizable Data via Contrastive Learning for Semantic Communications

Christina Chaccour, Walid Saad

Abstract

Achieving artificially intelligent-native wireless networks is necessary for the operation of future 6G applications such as the metaverse. Nonetheless, current communication schemes are, at heart, a mere reconstruction process that lacks reasoning. One key solution that enables evolving wireless communication to a human-like conversation is semantic communications. In this paper, a novel machine reasoning framework is proposed to pre-process and disentangle source data so as to make it semantic-ready. In particular, a novel contrastive learning framework is proposed, whereby instance and cluster discrimination are performed on the data. These two tasks enable increasing the cohesiveness between data points mapping to semantically similar content elements and disentangling data points of semantically different content elements. Subsequently, the semantic deep clusters formed are ranked according to their level of confidence. Deep semantic clusters of highest confidence are considered learnable, semantic-rich data, i.e., data that can be used to build a language in a semantic communications system. The least confident ones are considered, random, semantic-poor, and memorizable data that must be transmitted classically. Our simulation results showcase the superiority of our contrastive learning approach in terms of semantic impact and minimalism. In fact, the length of the semantic representation achieved is minimized by 57.22% compared to vanilla semantic communication systems, thus achieving minimalist semantic representations.

The Evolution of Wireless Communications

As we move toward the 6G era, the paradigm of wireless communication is shifting from "classical" bit-level reconstruction to "semantic" communications. While classical systems focus on perfectly reconstructing every bit transmitted, semantic communications aim to convey the underlying meaning of a message, much like a human conversation. This transition is essential for high-bandwidth, low-latency applications like the metaverse, where reasoning-driven radio nodes can significantly reduce the burden on the network core.

Comparison between classical bit-reconstruction and semantic reasoning-driven communication.

In this new framework, communication is viewed as a knowledge transfer between a "teacher" (transmitter) and an "apprentice" (receiver) who share a common, evolving language. However, current AI-driven models often struggle to distinguish between data that is rich in meaning and data that is simply random noise.

The Challenge of Entangled Data

Raw data, such as video or sensor feeds, is inherently "entangled." It contains both meaningful structural information and spurious, random noise. If a semantic communication system attempts to "learn" a language based on this noise, the resulting language becomes overly complex and inefficient, leading to overfitting. To address this, data must be pre-processed to be "semantic-ready" by disentangling learnable patterns from memorizable noise.

Proposed Contrastive Learning Framework

The proposed framework utilizes Contrastive Learning to distinguish between similar and dissimilar data points without explicit labels. The process involves two main stages:

  1. Instance Discrimination: The system creates perturbed versions of a data point (e.g., rotating an image) and trains a model to recognize that these versions represent the same underlying instance, building robustness against environmental shifts.
  2. Cluster Discrimination: Data points are grouped into "deep semantic clusters." The goal is to maximize the cohesiveness of points sharing the same meaning while pushing apart those with different meanings.

Following clustering, the framework ranks clusters based on confidence. High-confidence clusters are labeled as "learnable" (semantic-rich) and used to build the semantic language. Low-confidence clusters are identified as "memorizable" (semantic-poor) noise and are designated for transmission via classical bit-level methods.

Key Results and Performance

The framework was evaluated using binary equivalents of standard datasets like CIFAR-10, CIFAR-100, and ImageNet-10/Dogs. The results demonstrate that by filtering out "memorizable" data, the system maintains a lean and efficient semantic dictionary.

Average representation length vs. content complexity, highlighting the efficiency of the proposed disentanglement method.

Key performance metrics include:

  • Minimalist Representations: The average length of the semantic representation was reduced by 57.22% compared to vanilla semantic systems.
  • Increased Efficiency: The "semantic impact"—measuring computational savings at the receiver—was 71.9% higher than conventional approaches.
  • Scalability: As content complexity increases, the proposed method prevents the "complexity explosion" seen in standard semantic models.

Conclusion

This research highlights the necessity of hybrid communication systems for 6G. By intelligently deciding which parts of a data stream represent "meaning" and which are merely "bits," networks can achieve unprecedented efficiency. Future work will focus on dynamic threshold setting for confidence levels and optimizing the computational costs of on-device contrastive learning.

Summary updated

Abstract

Achieving artificially intelligent-native wireless networks is necessary for the operation of future 6G applications such as the metaverse. Nonetheless, current communication schemes are, at heart, a mere reconstruction process that lacks reasoning. One key solution that enables evolving wireless communication to a human-like conversation is semantic communications. In this paper, a novel machine reasoning framework is proposed to pre-process and disentangle source data so as to make it semantic-ready. In particular, a novel contrastive learning framework is proposed, whereby instance and cluster discrimination are performed on the data. These two tasks enable increasing the cohesiveness between data points mapping to semantically similar content elements and disentangling data points of semantically different content elements. Subsequently, the semantic deep clusters formed are ranked according to their level of confidence. Deep semantic clusters of highest confidence are considered learnable, semantic-rich data, i.e., data that can be used to build a language in a semantic communications system. The least confident ones are considered, random, semantic-poor, and memorizable data that must be transmitted classically. Our simulation results showcase the superiority of our contrastive learning approach in terms of semantic impact and minimalism. In fact, the length of the semantic representation achieved is minimized by 57.22% compared to vanilla semantic communication systems, thus achieving minimalist semantic representations.
BibTeX
@inproceedings{chaccour2022disentangling,
  author = {Christina Chaccour and Walid Saad},
  title = {{Disentangling Learnable and Memorizable Data via Contrastive Learning for Semantic Communications}},
  booktitle = {IEEECONF},
  year = {2022},
  url = {https://openreview.net/forum?id=i8XZdiXt5rz},
}
Figure