CrowdForge: Effects of Audio and Visual Audience Participation Modalities on Agency, Collaboration, and Engagement in AI-Mediated MR Performances

Torin Hopkins, Shih-Yu Ma, Zander Trafton-Hancock, Ellen Yi-Luen Do, Luca TurchetarXiv 2026 · arXiv

TL;DR

CrowdForge is an AI and Mixed Reality (MR) framework that allows live audiences to collectively steer musical and visual elements of a performance. The study finds that observability—the immediate, recognizable feedback of an action—is the single most important factor in making an audience feel they have "agency" in a high-tech performance.

What Problem Does This Paper Address?

Traditional live music is a one-way street: the artist performs, and the audience listens. While Technology-Mediated Audience Participation (TMAP) aims to break this wall, it often struggles with a "black box" problem. If an audience member votes for a "funky" vibe, but the resulting AI-generated music just sounds like a jumbled mess, they don't feel like they’ve actually influenced anything.

The authors of CrowdForge set out to understand how different types of participation—audio vs. visual—impact an audience’s sense of agency, engagement, and social connection. They specifically look at how generative AI and MR can be used to balance accessible interaction with meaningful creative contribution.

Methods

The researchers developed CrowdForge, a networked system that integrates several cutting-edge components:

  • Web Interface: A mobile-friendly portal where audience members vote on parameters like musical genre, tempo, instrument effects, and virtual environment skins.
  • AI Music Engine: A domain-adapted version of the Llama-3.2-1B model (dubbed Llama-Z) that generates multitrack MIDI in real-time based on audience input.
  • Performance Pipeline: The AI-generated MIDI is fed into Ableton Live, where a DJ manipulates the tracks while a live guitarist improvises over them.
  • MR Environment: Built in Unity 3D, this virtual stage features avatars and dancers that respond to the music and audience selections, projected for the crowd to see.

The study involved 19 participants across three conditions: Audio-only, Visual-only, and Both. The team used a mixed-methods approach, combining behavioral data (tracking what people actually clicked), psychometric questionnaires (like the Self-Assessment Manikin), and semi-structured interviews.

System diagram and signal flow illustrating audience control over the performance.

Datasets

To train the Llama-Z model for multitrack music generation, the authors used:

  • Los Angeles MIDI Dataset (LAMD): 20,000 sampled segments.
  • Slakh2100 dataset: Used for additional multitrack variety.
  • Normalization: All data was normalized to a 4/4 time signature and mathematically shifted to C-Major or A-Minor to ensure harmonic coherence during live generation.

Key Insights and Ideas

  • Observability is the Primary Driver of Agency: Participants didn't care how many buttons they had; they cared if they could see or hear the result. Visual changes (like changing the environment to a "Mars scape") were perceived as more controllable because they were immediately obvious.
  • The "Productive Adversarial" Dynamic: Interestingly, audiences didn't always try to make "good" music. Instead, they often chose "intentionally wonky" or extreme combinations to test the system's limits and see how the human guitarist would react. This created a "push towards chaos" that actually deepened engagement.
  • The Illusion of Choice: Granular AI settings (like specific effects for a virtual flute) had high "skip rates" (up to 38.5%). Audiences preferred macro-structural changes—like tempo and genre—that had a clear, high-impact effect on the performance.
  • Sensory Asymmetry: Audio is great for driving emotional arousal and valence, but it’s prone to "structural conflict" (clashing sounds). Visuals, however, can coexist without clashing, providing a clear "receipt" for the audience's participation.
Overview of the audience participation workflow and resulting MR environment.

Key Results

  • Emotional Impact: Both the "Audio" and "Both" conditions significantly elevated Valence (pleasure) and Arousal compared to a neutral baseline. The "Visual-only" condition did not have the same emotional "oomph."
  • Agency Scores: "Guitar tone for human player" was the only audio parameter that significantly boosted the sense of agency (p = 0.007). This suggests that influencing a human performer feels more meaningful than influencing an AI.
  • Social Presence: Adding audio to a visual experience significantly enhanced the feeling of being "connected" to other audience members (p = 0.015).
Comparison of dominance, valence, and arousal scores across conditions.

Limitations and Future Work

The study was limited by a relatively small sample size (N=19) and an asynchronous voting system (voting happened before the set, not during). The authors also noted that the visuals didn't actually influence the human guitarist's playing—only the audience saw them.

Future directions include: * Real-time Interaction: Moving from "episodic" voting to continuous, real-time "world-building." * Closing the Loop: Having human dancers respond to the MR visuals, making the visual modality as impactful on the performers as the audio is. * Remote Participation: Testing how this system scales for virtual or hybrid audiences in the "Musical Metaverse."

Perceived influence and agency ratings for specific audio elements.

Summary updated

Abstract

Technology-mediated audience participation (TMAP) enables audiences to influence live performances, yet balancing interaction with meaningful agency remains challenging. While AI-generated music and MR enable new forms of audio and visual audience participation, little is known about how these participation modalities influence audience experiences during live musical performance. In particular, it remains unclear how they shape perceived agency, collaboration, and engagement, and how they complement one another within TMAP systems. We present CrowdForge, an AI- and MR-mediated paradigm where audiences collectively control generative music and virtual environments during a live performance featuring a guitarist and DJ. Through a mixedmethods concert study evaluating three interaction conditions (audio, visual, and combined audio-visual), we demonstrate that observability is the primary driver of audience agency. While visual interaction provided easily observable feedback, audio parameters offered macro-structural control over the music. Audiences tended to use the control enabled by AI audio generation to test musical boundaries, introducing a productive adversarial tension between the crowd and the musicians. This dynamic pushed creative limits, deepened engagement, and generated moments of musical coherence when performers successfully resolved audience-induced challenges. We argue future TMAP systems should prioritize human-centered, highly observable interactions to build agency, while combining the expressiveness of music with the clarity of visual feedback.
BibTeX
@misc{hopkins2026crowdforge,
  author = {Torin Hopkins and Shih-Yu Ma and Zander Trafton-Hancock and Ellen Yi-Luen Do and Luca Turchet},
  title = {{CrowdForge: Effects of Audio and Visual Audience Participation Modalities on Agency, Collaboration, and Engagement in AI-Mediated MR Performances}},
  year = {2026},
  eprint = {2611.03389},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.03389},
}
Figure