MUSIC COLLAB: An IoT and ML Based Solution for Remote Music Collaboration (Student Abstract)

Nishtha Nayar, Divya LohaniAAAI 2020 · DOI

Introduction

Remote music collaboration, often considered the "holy grail" for geographically separated musicians, remains a significant technical challenge. In live jamming sessions, even minor network instabilities—such as jitter or packet loss—can disrupt the rhythmic flow and creative synergy. Existing tools often demand high-bandwidth, low-latency connections beyond the reach of average users. This research proposes a dual-layered solution: optimizing IoT network protocols for stability and employing machine learning to predict and "fill" missing audio segments in real-time.

Methodology

The study investigates two primary components: network transport reliability and predictive generative modeling.

Network Layer Optimization

Using WireShark, the researchers analyzed server-client communication across three scenarios: server-to-client MP3 requests, client-to-server MP3 uploads, and multi-client synchronization. The performance of Transmission Control Protocol (TCP) was compared against User Datagram Protocol (UDP) to determine which offers the most consistent performance for audio data in an IoT context.

Predictive Modeling for Packet Loss

To mitigate the impact of lost audio packets, the study utilized Google’s Magenta framework to train three variants of Melody Recurrent Neural Networks (RNNs):

  • Basic RNN: A standard recurrent architecture.
  • Lookback RNN: Designed to maintain structural repetition by referencing previous notes.
  • Attention RNN: Utilizes attention mechanisms to capture long-term melodic dependencies.

These models were trained on a dataset of approximately 11,000 MIDI files encompassing various instruments. The objective was to generate a 3-bar "fill" based on a 5-bar primer, acting as a safety net during network dropouts.

Network Performance Analysis

Contrary to the traditional preference for UDP in real-time media due to its lower overhead, the researchers found that TCP provided more consistent and measurable performance data in their IoT environment.

Round Trip Time (RTT) analysis for network stability assessment.

As shown in the RTT analysis, TCP allowed for stable monitoring of latency, which is crucial for maintaining synchronization in a collaborative musical environment. While throughput varied, the ability to track Round Trip Time provided a more reliable baseline for synchronization than the "fire-and-forget" nature of UDP.

Model Performance and Evaluation

The quantitative evaluation of the RNN models revealed a "musicality paradox." While the Basic RNN achieved the lowest loss and highest training accuracy, qualitative assessments by musicians favored the more complex models.

Comparative performance metrics for Basic, Lookback, and Attention RNN models.

The data in the table above highlights that while the Attention and Lookback models had slightly higher loss values, they were more effective at capturing the nuances of musical structure. The Basic RNN, despite its superior technical metrics, tended to produce repetitive and less artistically viable melodies.

Discussion

The study suggests that generative AI can serve as a real-time "patch" for network failures. Instead of merely using AI for composition, it can be deployed as a fail-safe mechanism to prevent silence during transmission errors. By "hallucinating" missing notes, the system maintains the rhythmic momentum of a performance even when the network falters. Furthermore, the results emphasize that standard machine learning metrics like loss and accuracy do not always align with artistic quality; the Attention RNN, despite lower technical scores, produced more coherent and "musical" results.

Conclusion and Future Work

The research demonstrates that a combination of reliable transport protocols and predictive modeling can significantly enhance remote music collaboration. Future work will focus on integrating RNN prediction directly into live audio streams and conducting formal user studies to evaluate the subjective experience of musicians using AI-assisted "gap-filling" during real-time sessions.

Summary updated

BibTeX
@inproceedings{nayar2020music,
  author = {Nishtha Nayar and Divya Lohani},
  title = {{MUSIC COLLAB: An IoT and ML Based Solution for Remote Music Collaboration (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i10.7214},
}
Figure