When Music Needs Meaning: Semantic Communication as a Transformative Enabler for the Musical Metaverse

Ali Al Housseini, Jaime Llorca, Cristina Rottondi, Luca Turchet, Tiziano Leidi, Omran AyoubarXiv 2026 · arXiv

This paper proposes a shift in how we think about remote musical collaboration. As we move toward a "Musical Metaverse"—where musicians, AI agents, and audiences interact in immersive, multi-modal environments—traditional methods of sending raw audio and video data are hitting a wall. The authors argue that the solution lies in semantic communication: transmitting the meaning of music rather than just its raw signal.

TL;DR

The authors propose a three-pillar framework for a "Musical Metaverse" that uses semantic communication to overcome the extreme latency and bandwidth hurdles of real-time remote performance. By transmitting musically meaningful features (like pitch and rhythm) instead of raw samples, the system can prioritize what matters most for human coordination and perception in 6G networks.

What Problem Does This Paper Address?

Networked Music Performance (NMP) is notoriously difficult. To play together naturally, musicians need end-to-end audio latency under 40ms, and haptic feedback (like the feel of a drum hit) requires sub-10ms responsiveness.

Current systems follow Shannon’s paradigm: they focus on the reliable delivery of bits, treating all data in a stream with equal importance. However, in a complex "Musical Metaverse" (MM) involving audio, 3D avatars, and haptic suits, this approach doesn't scale. A drummer needs to hear a precise rhythmic onset, but they might not care about the high-fidelity reverberation of a background guitar. The "Shannon gap" is the inability of current networks to distinguish between these musically critical events and secondary details, leading to congestion and broken synchronization.

Methods

The authors propose a semantic-first reference architecture built on three pillars:

  1. Musical Semantics Extraction (P1): Device-side encoders use AI (like neural audio codecs) to extract structured descriptors from raw signals. These are categorized into levels: L2 (Features) like pitch and onset, L3 (Structure) like chord progressions, and L4 (Intent) like emotional valence.
  2. Semantic-aware Orchestration (P2): An edge-computing layer acts as a "smart router." It looks at a receiver's specific task (e.g., "I am the vocalist, I need pitch cues") and the current network state to decide which semantic fields to send and at what resolution.
  3. Cross-modal Semantic Synchronization (P3): This layer aligns different modalities (audio, haptic, visual) using a shared musical time base (beats and bars) rather than just absolute network timestamps.

![Fig.

  1. Proposed semantic-first architecture showing the flow from extraction to orchestration and synchronization.](/figures/art_72e289a5fd9544d8909f64a97ed6e74f)

The paper illustrates this with a guitar phrase example, where a drummer receives a simplified "click" and haptic cue for a note, while a vocalist receives a detailed pitch contour for the same event.

![Fig.

  1. Illustrative semantic flow for a guitar phrase, showing how different receivers get different data based on their needs.](/figures/art_ebf47a9038824635bafbaba9c61b5f9b)

Datasets

As a position and framework paper, it does not introduce a new training dataset. Instead, it surveys and integrates existing components like:

  • Neural Audio Codecs: EnCodec, DAC, and MuCodec.
  • Music Representation Models: MERT, Stem-JEPA, and MuLan.
  • Protocols: OSC (Open Sound Control) and Ableton Link for timing.

The authors do, however, propose new diagnostic metrics for future benchmarking:

  • Semantic Musical Fidelity (SMF): Measures how well specific required musical features survive transmission.
  • Ensemble Synchronization Index (ESI): Evaluates the relative alignment of multi-modal events across different sites.
  • Emotional Coherence Score (ECS): An exploratory metric to see if inferred emotional trajectories (like "tension") remain consistent across audio, visual, and kinematic streams.

Key Insights and Ideas

  • Music is a Demanding Edge Case: Unlike generic video conferencing, music relies on a shared temporal reference (the beat) and a perceptual hierarchy where rhythm often trumps timbral detail.
  • Task-Dependent Routing: The "meaning" of a data stream changes depending on who is receiving it. A sound engineer needs timbre; a drummer needs timing. Semantic communication allows the network to adapt to these roles dynamically.
  • Creative Adaptation: The authors highlight that musicians are not mechanical systems; they actively anticipate and compensate for network lag. A semantic system can support this by providing "intent" cues before the full audio even arrives.

Key Results

The paper is a theoretical framework and roadmap, so it does not report empirical performance numbers (e.g., "X% reduction in latency"). Instead, its "results" are the formal definitions and the architecture itself.

The authors define three primary use cases for the Musical Metaverse, each with distinct requirements:

  • UC1: Remote Jamming (Small ensemble, ultra-low latency).
  • UC2: Virtual Concert (Large audience, scalable distribution).
  • UC3: AI Co-creation (Human-AI interaction, exchanging intent).
Table I. Representative MM Use Cases and their specific requirements.

Limitations and Future Work

The authors openly acknowledge that this framework is currently unvalidated. The primary limitation is the lack of empirical testing to prove that semantic extraction doesn't add more processing delay than the transmission time it saves.

Future Work (The Roadmap): 1. Stage 1: Implement a "bounded task" (like a simple remote guitar monitor) to measure the actual computational costs of extraction and rendering. 2. Stage 2: Compare the semantic approach against standard audio codecs and priority-aware networking to see if the "semantic gain" is real. 3. Stage 3: Conduct human-in-the-loop studies with professional musicians to see if these technical metrics (like ESI) actually correlate with a better playing experience.

Summary updated

Abstract

The Musical Metaverse (MM) enables geographically distributed musicians, audiences, sound engineers, and AI agents to interact through synchronized audio, visual, kinematic, and haptic streams, imposing latency, synchronization, bandwidth, and scalability requirements that are difficult to satisfy with conventional bit-level communication. This paper argues that semantic communication, understood here as the transmission of musically meaningful information rather than raw audio-visual-haptic samples, provides a principled communication paradigm for the MM. We define the MM from a communication perspective and derive modality-specific require ments; propose a three-pillar framework for musical semantic communication comprising semantics extraction, semantic-aware orchestration, and cross-modal semantic synchronization; survey representative work across semantic communication, music AI and edge orchestration, identifying the gaps that prevent their integration; and propose a semantic-first reference architecture based on device-side semantic encoders, edge semantic routers, and cross-modal semantic synchronization. The resulting framework positions the MM as a concrete and demanding use case for 6G semantic communication.
BibTeX
@misc{housseini2026when,
  author = {Ali Al Housseini and Jaime Llorca and Cristina Rottondi and Luca Turchet and Tiziano Leidi and Omran Ayoub},
  title = {{When Music Needs Meaning: Semantic Communication as a Transformative Enabler for the Musical Metaverse}},
  year = {2026},
  eprint = {2611.03995},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.03995},
}
Figure