Abstract
The SEND/RETURN (S/R) project explores the efficacy of content-based music recommendations alongside a uniquely generated Unreal Engine 5 (UE5) virtual environment based on audio features. S/R employs both a k-means clustering algorithm using audio features and a fast pattern matching (FPM) algorithm using 30-second audio signals to find similar-sounding songs to recommend to users. The feature values of the recommended song are then communicated via HTTP to the UE5 virtual environment, which changes a number of effects in real-time. All of this is being replicated from a listen-server to other clients to create a multiplayer audio session. S/R successfully creates a lightweight online environment that replicates song information to all clients and suggests new songs that alter the world around you. In this work, we extend S/R by training a convolutional neural network using Mel-spectrograms of 30-second audio samples to predict the mood of a song. The developed convolutional model had a validation accuracy of 67.5% in predicting 4 moods ('calm', 'energetic', 'happy', 'sad').
Introduction
Traditional Music Recommender Systems (MRS) often suffer from "popularity bias," where algorithms favor mainstream hits over niche content because they rely on collaborative filtering. Furthermore, music discovery is typically a solitary, UI-driven experience. The SEND/RETURN (S/R) project addresses these gaps by moving toward content-based discovery—recommending music based on how it actually sounds—and creating an immersive social experience. In S/R, music isn't just heard; it is experienced visually in a shared virtual space where the environment reacts to audio features in real-time.
System Architecture and Methods
The S/R system integrates several distinct technologies to create its multi-user experience. At its core is a "listen-server" architecture built in Unreal Engine 5 (UE5). A host (P1) searches for music, and the resulting audio and visual effects are replicated across all connected clients (P2, P3) in real-time.
The system utilizes two primary recommendation engines:
- k-means clustering: Groups songs based on pre-calculated Spotify metadata such as tempo and danceability.
- Fast Pattern Matching (FPM): Uses the STUMPY library to analyze raw 30-second audio signals, looking for structural similarities in the waveform.
To drive environmental changes, the authors developed a 2D Convolutional Neural Network (CNN) for mood classification. This model takes Mel-spectrograms—visual representations of audio that mimic human hearing—as input to categorize songs into four moods: Calm, Energetic, Happy, and Sad.
Visual Mapping and Results
Audio features and predicted moods are mapped directly to UE5 parameters. For instance, "Energy" controls firework projectiles, "Tempo" dictates post-processing effects, and the predicted "Mood" changes the color of the virtual moon.
Key results from the project include:
- Model Performance: The custom CNN achieved a 67.5% validation accuracy across the four mood categories. Notably, the model was exceptionally good at identifying "Calm" music (0.98 precision) due to its distinct, stable visual patterns in spectrograms. It struggled most with "Happy" vs. "Energetic" songs, which share high "arousal" levels.
- Recommendation Quality: Users preferred recommendations from the FPM algorithm (raw signal analysis) over k-means (metadata analysis), suggesting that mathematical patterns in the audio signal capture a song's "vibe" more effectively than high-level descriptors.
- User Experience: In VR testing, participants found the environment-altering effects, such as the "Blueprint Shader" tied to time signatures, to be a unique and pleasant way to experience music discovery.
Conclusion and Future Work
While S/R successfully creates a lightweight, synchronized environment, several areas for improvement remain. The current dataset of ~1,300 songs is relatively small; expanding it could improve the CNN's robustness. Future work may also involve using Siamese Neural Networks to better discern subtle differences between high-arousal moods like "Happy" and "Energetic." Additionally, incorporating lyrical analysis via Natural Language Processing (NLP) could resolve ambiguities in mood detection, and future AI models might eventually "auto-generate" these visual environments based on broader contextual data.
Summary updated