This paper proposes a shift in how we think about remote musical collaboration. As we move toward a "Musical Metaverse"—where musicians, AI agents, and audiences interact in immersive, multi-modal environments—traditional methods of sending raw audio and video data are hitting a wall. The authors argue that the solution lies in semantic communication: transmitting the meaning of music rather than just its raw signal.
TL;DR
The authors propose a three-pillar framework for a "Musical Metaverse" that uses semantic communication to overcome the extreme latency and bandwidth hurdles of real-time remote performance. By transmitting musically meaningful features (like pitch and rhythm) instead of raw samples, the system can prioritize what matters most for human coordination and perception in 6G networks.
What Problem Does This Paper Address?
Networked Music Performance (NMP) is notoriously difficult. To play together naturally, musicians need end-to-end audio latency under 40ms, and haptic feedback (like the feel of a drum hit) requires sub-10ms responsiveness.
Current systems follow Shannon’s paradigm: they focus on the reliable delivery of bits, treating all data in a stream with equal importance. However, in a complex "Musical Metaverse" (MM) involving audio, 3D avatars, and haptic suits, this approach doesn't scale. A drummer needs to hear a precise rhythmic onset, but they might not care about the high-fidelity reverberation of a background guitar. The "Shannon gap" is the inability of current networks to distinguish between these musically critical events and secondary details, leading to congestion and broken synchronization.
Methods
The authors propose a semantic-first reference architecture built on three pillars:
- Musical Semantics Extraction (P1): Device-side encoders use AI (like neural audio codecs) to extract structured descriptors from raw signals. These are categorized into levels: L2 (Features) like pitch and onset, L3 (Structure) like chord progressions, and L4 (Intent) like emotional valence.
- Semantic-aware Orchestration (P2): An edge-computing layer acts as a "smart router." It looks at a receiver's specific task (e.g., "I am the vocalist, I need pitch cues") and the current network state to decide which semantic fields to send and at what resolution.
- Cross-modal Semantic Synchronization (P3): This layer aligns different modalities (audio, haptic, visual) using a shared musical time base (beats and bars) rather than just absolute network timestamps.

The paper illustrates this with a guitar phrase example, where a drummer receives a simplified "click" and haptic cue for a note, while a vocalist receives a detailed pitch contour for the same event.

Datasets
As a position and framework paper, it does not introduce a new training dataset. Instead, it surveys and integrates existing components like:
- Neural Audio Codecs: EnCodec, DAC, and MuCodec.
- Music Representation Models: MERT, Stem-JEPA, and MuLan.
- Protocols: OSC (Open Sound Control) and Ableton Link for timing.
The authors do, however, propose new diagnostic metrics for future benchmarking:
- Semantic Musical Fidelity (SMF): Measures how well specific required musical features survive transmission.
- Ensemble Synchronization Index (ESI): Evaluates the relative alignment of multi-modal events across different sites.
- Emotional Coherence Score (ECS): An exploratory metric to see if inferred emotional trajectories (like "tension") remain consistent across audio, visual, and kinematic streams.
Key Insights and Ideas
- Music is a Demanding Edge Case: Unlike generic video conferencing, music relies on a shared temporal reference (the beat) and a perceptual hierarchy where rhythm often trumps timbral detail.
- Task-Dependent Routing: The "meaning" of a data stream changes depending on who is receiving it. A sound engineer needs timbre; a drummer needs timing. Semantic communication allows the network to adapt to these roles dynamically.
- Creative Adaptation: The authors highlight that musicians are not mechanical systems; they actively anticipate and compensate for network lag. A semantic system can support this by providing "intent" cues before the full audio even arrives.
Key Results
The paper is a theoretical framework and roadmap, so it does not report empirical performance numbers (e.g., "X% reduction in latency"). Instead, its "results" are the formal definitions and the architecture itself.
The authors define three primary use cases for the Musical Metaverse, each with distinct requirements:
- UC1: Remote Jamming (Small ensemble, ultra-low latency).
- UC2: Virtual Concert (Large audience, scalable distribution).
- UC3: AI Co-creation (Human-AI interaction, exchanging intent).
Limitations and Future Work
The authors openly acknowledge that this framework is currently unvalidated. The primary limitation is the lack of empirical testing to prove that semantic extraction doesn't add more processing delay than the transmission time it saves.
Future Work (The Roadmap): 1. Stage 1: Implement a "bounded task" (like a simple remote guitar monitor) to measure the actual computational costs of extraction and rendering. 2. Stage 2: Compare the semantic approach against standard audio codecs and priority-aware networking to see if the "semantic gain" is real. 3. Stage 3: Conduct human-in-the-loop studies with professional musicians to see if these technical metrics (like ESI) actually correlate with a better playing experience.
Summary updated