TL;DR
CrowdForge is an AI and Mixed Reality (MR) framework that allows live audiences to collectively steer musical and visual elements of a performance. The study finds that observability—the immediate, recognizable feedback of an action—is the single most important factor in making an audience feel they have "agency" in a high-tech performance.
What Problem Does This Paper Address?
Traditional live music is a one-way street: the artist performs, and the audience listens. While Technology-Mediated Audience Participation (TMAP) aims to break this wall, it often struggles with a "black box" problem. If an audience member votes for a "funky" vibe, but the resulting AI-generated music just sounds like a jumbled mess, they don't feel like they’ve actually influenced anything.
The authors of CrowdForge set out to understand how different types of participation—audio vs. visual—impact an audience’s sense of agency, engagement, and social connection. They specifically look at how generative AI and MR can be used to balance accessible interaction with meaningful creative contribution.
Methods
The researchers developed CrowdForge, a networked system that integrates several cutting-edge components:
- Web Interface: A mobile-friendly portal where audience members vote on parameters like musical genre, tempo, instrument effects, and virtual environment skins.
- AI Music Engine: A domain-adapted version of the Llama-3.2-1B model (dubbed Llama-Z) that generates multitrack MIDI in real-time based on audience input.
- Performance Pipeline: The AI-generated MIDI is fed into Ableton Live, where a DJ manipulates the tracks while a live guitarist improvises over them.
- MR Environment: Built in Unity 3D, this virtual stage features avatars and dancers that respond to the music and audience selections, projected for the crowd to see.
The study involved 19 participants across three conditions: Audio-only, Visual-only, and Both. The team used a mixed-methods approach, combining behavioral data (tracking what people actually clicked), psychometric questionnaires (like the Self-Assessment Manikin), and semi-structured interviews.
Datasets
To train the Llama-Z model for multitrack music generation, the authors used:
- Los Angeles MIDI Dataset (LAMD): 20,000 sampled segments.
- Slakh2100 dataset: Used for additional multitrack variety.
- Normalization: All data was normalized to a 4/4 time signature and mathematically shifted to C-Major or A-Minor to ensure harmonic coherence during live generation.
Key Insights and Ideas
- Observability is the Primary Driver of Agency: Participants didn't care how many buttons they had; they cared if they could see or hear the result. Visual changes (like changing the environment to a "Mars scape") were perceived as more controllable because they were immediately obvious.
- The "Productive Adversarial" Dynamic: Interestingly, audiences didn't always try to make "good" music. Instead, they often chose "intentionally wonky" or extreme combinations to test the system's limits and see how the human guitarist would react. This created a "push towards chaos" that actually deepened engagement.
- The Illusion of Choice: Granular AI settings (like specific effects for a virtual flute) had high "skip rates" (up to 38.5%). Audiences preferred macro-structural changes—like tempo and genre—that had a clear, high-impact effect on the performance.
- Sensory Asymmetry: Audio is great for driving emotional arousal and valence, but it’s prone to "structural conflict" (clashing sounds). Visuals, however, can coexist without clashing, providing a clear "receipt" for the audience's participation.
Key Results
- Emotional Impact: Both the "Audio" and "Both" conditions significantly elevated Valence (pleasure) and Arousal compared to a neutral baseline. The "Visual-only" condition did not have the same emotional "oomph."
- Agency Scores: "Guitar tone for human player" was the only audio parameter that significantly boosted the sense of agency (p = 0.007). This suggests that influencing a human performer feels more meaningful than influencing an AI.
- Social Presence: Adding audio to a visual experience significantly enhanced the feeling of being "connected" to other audience members (p = 0.015).
Limitations and Future Work
The study was limited by a relatively small sample size (N=19) and an asynchronous voting system (voting happened before the set, not during). The authors also noted that the visuals didn't actually influence the human guitarist's playing—only the audience saw them.
Future directions include: * Real-time Interaction: Moving from "episodic" voting to continuous, real-time "world-building." * Closing the Loop: Having human dancers respond to the MR visuals, making the visual modality as impactful on the performers as the audio is. * Remote Participation: Testing how this system scales for virtual or hybrid audiences in the "Musical Metaverse."
Summary updated