Multimodal Trustworthy Semantic Communication for Audio-Visual Event Localization

Yi Li, Zhe Xiang, F. Richard Yu, Zhangshuang Guan, Hui Ji, Zhiguo Wan, Cheng FengarXiv 2024 · DOI · arXiv

Abstract: The exponential growth in wireless data traffic, driven by the proliferation of mobile devices and smart applications, poses significant challenges for modern communication systems. Ensuring the secure and reliable transmission of multimodal semantic information is increasingly critical, particularly for tasks like Audio-Visual Event (AVE) localization. This letter introduces MMTrustSC, a novel framework designed to address these challenges by enhancing the security and reliability of multimodal communication. MMTrustSC incorporates advanced semantic encoding techniques to safeguard data integrity and privacy. It features a two-level coding scheme that combines error-correcting codes with conventional encoders to improve the accuracy and reliability of multimodal data transmission. Additionally, MMTrustSC employs hybrid encryption, integrating both asymmetric and symmetric encryption methods, to secure semantic information and ensure its confidentiality and integrity across potentially hostile networks. Simulation results validate MMTrustSC's effectiveness, demonstrating substantial improvements in data transmission accuracy and reliability for AVE localization tasks. This framework represents a significant advancement in managing intermodal information complementarity and mitigating physical noise, thus enhancing overall system performance.

Introduction

Traditional communication systems prioritize the perfect transmission of bits. In contrast, semantic communication focuses on transmitting the "meaning" of data, offering greater efficiency and robustness to noise. However, security remains a major hurdle. Standard encryption algorithms are notoriously fragile; a single bit error during transmission can cause decryption to fail entirely. This is particularly problematic in semantic communication, where data often resides in the real-number domain. Furthermore, multimodal tasks like Audio-Visual Event (AVE) localization—identifying a specific event, such as a dog barking, in both time and space—suffer from inter-modal interference where noise in one stream confuses the processing of another.

To address these gaps, we present MMTrustSC, a framework that integrates "trustworthiness"—security and reliability—into multimodal semantic communication.

The MMTrustSC Framework

MMTrustSC employs a multi-layered architecture to ensure that semantic information remains private and intact, even in highly noisy wireless environments.

The MMTrustSC system architecture, illustrating the transmitter and receiver components for multimodal data.

Two-Level Coding and Hybrid Encryption

To solve the "fragile encryption" problem, MMTrustSC uses a nested approach. It applies conventional channel encoders wrapped in an outer layer of Reed-Solomon (RS) error-correcting codes. This creates a safety net, allowing the receiver to correct minor bit errors before decryption occurs.

For security, the system utilizes Hybrid Encryption:

  • Elliptic Curve Cryptography (ECC): An asymmetric method used for secure session key exchange.
  • AES-CBC: A symmetric method used for the high-speed transmission of the actual semantic features.

Cross-Modal Semantic Encoding

The framework features an Attention-Guided Visual-Audio (AGVA) mechanism. Instead of processing audio and video separately, it uses recovered audio signals to guide the visual encoder, focusing its attention on relevant spatial regions (e.g., the source of a sound). The system leverages pre-trained VGG-19 networks for feature extraction and LSTMs for temporal sequence modeling to capture the evolution of events over time.

Experimental Results

The framework was evaluated using the AVE subset of the AudioSet dataset, which includes 4,143 videos across 28 diverse event types. Each video contains specific temporal annotations for localization.

Performance Across Channels

MMTrustSC was tested across three channel types: AWGN (standard noise), Rayleigh (multipath fading), and Rician (line-of-sight fading).

Accuracy of MMTrustSC compared to baseline models across AWGN, Rayleigh, and Rician channels at varying SNR levels.

As shown in the performance plots, MMTrustSC significantly outperforms traditional methods (JPEG/PCM) and single-modality baselines. At a very noisy Signal-to-Noise Ratio (SNR) of 0dB in a Rician channel, the trustworthy model achieved an accuracy of 0.284, compared to only 0.184 for the version without the error-correction/encryption layer. In cleaner channels (SNR > 15dB), the system reaches near-optimal localization accuracy (approx. 0.73), matching or exceeding state-of-the-art non-secure models while providing full encryption.

Key Insights

  • The Decryption-Noise Paradox: By placing RS codes after encryption but before the physical channel, the framework allows standard cryptographic tools to function in noisy, analog-like environments.
  • Semantic vs. Channel Noise: The framework distinguishes between loss of meaning during encoding (semantic noise) and bit errors during transmission (channel noise), providing specific mechanisms (LSTMs and RS codes, respectively) to mitigate both.
  • Audio as a Spatial Filter: Qualitative analysis confirms that audio acts as a spatial filter for video, significantly improving localization accuracy by helping the model ignore irrelevant background visual noise.

Conclusion

MMTrustSC demonstrates that security and semantic efficiency are not mutually exclusive. By combining hybrid encryption with robust error correction, the framework enables accurate and private multimodal communication. Future work will explore reducing computational overhead for low-power IoT devices and testing the framework's resilience against active adversarial attacks like jamming or semantic spoofing.

Summary updated

Abstract

The exponential growth in wireless data traffic, driven by the proliferation of mobile devices and smart applications, poses significant challenges for modern communication systems. Ensuring the secure and reliable transmission of multimodal semantic information is increasingly critical, particularly for tasks like Audio-Visual Event (AVE) localization. This letter introduces MMTrustSC, a novel framework designed to address these challenges by enhancing the security and reliability of multimodal communication. MMTrustSC incorporates advanced semantic encoding techniques to safeguard data integrity and privacy. It features a two-level coding scheme that combines error-correcting codes with conventional encoders to improve the accuracy and reliability of multimodal data transmission. Additionally, MMTrustSC employs hybrid encryption, integrating both asymmetric and symmetric encryption methods, to secure semantic information and ensure its confidentiality and integrity across potentially hostile networks. Simulation results validate MMTrustSC's effectiveness, demonstrating substantial improvements in data transmission accuracy and reliability for AVE localization tasks. This framework represents a significant advancement in managing intermodal information complementarity and mitigating physical noise, thus enhancing overall system performance.
BibTeX
@misc{li2024multimodal,
  author = {Yi Li and Zhe Xiang and F. Richard Yu and Zhangshuang Guan and Hui Ji and Zhiguo Wan and Cheng Feng},
  title = {{Multimodal Trustworthy Semantic Communication for Audio-Visual Event Localization}},
  year = {2024},
  eprint = {2411.01991},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.01991},
}
Figure