Modeling Subjective Interpretations of Sound and Music

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2125 papers166 conceptsfrozen 2026-10-04

In traditional machine learning, we treat labels as objective facts. A picture of a cat is a cat, and a recording of a "C major" chord is a C major chord. However, as researchers move deeper into the nuances of music and environmental audio, this "ground truth" paradigm is beginning to fracture.

Across a wide range of tasks—from genre classification to audio captioning—it is becoming clear that the "correct" label for a sound is often a matter of perspective rather than a physical property of the signal. This research direction focuses on moving beyond static, single-label datasets toward systems that embrace the inherent ambiguity and subjectivity of human perception.

The Myth of the Single Ground Truth

The fundamental challenge in audio analysis is that human listeners rarely agree on what they are hearing. In the vocadito dataset, Rachel M Bittner and Gabriel Meseguer-Brocal found that even expert musicians agree on the discrete note boundaries of vocal performances only about 64% of the time. What one expert calls a distinct note, another might call a stylistic ornament.

This disagreement isn't "noise" to be filtered out; it is a reflection of how humans process sound. Daniel Müllensiefen and colleagues argue in Evaluating a Chord-Labelling Algorithm that traditional evaluation unfairly penalizes algorithms for selecting valid alternative interpretations. If two human experts can reasonably disagree on a chord progression or a genre, a model that chooses either one should be considered "correct."

Social Constructs and Perceptual Ambiguity

Why is audio so subjective? The papers suggest two primary drivers:

  1. Social Constructs: Categories like "genre" are not intrinsic acoustic properties. As Anders Meng notes in An Investigation of Feature Models for Music Genre Classification, genres are social agreements with fuzzy boundaries. This creates a "performance ceiling" for models; a machine cannot be more accurate than the humans who provide its training data.
  2. Acoustic Overlap: Different physical events can sound nearly identical. Swapnil Bhosale and colleagues point out in their work on s2vscore that distinct objects often produce acoustically similar noises. Without visual context, a listener might describe a sound as "wind" while another hears "static noise."

From Hard Labels to Human Consensus

To address this, researchers are shifting from "hard" classification to "soft" consensus-based models. Instead of training a model to predict a single label, systems are being designed to navigate multiple valid interpretations.

For example, the Harmonix Set, introduced by Matthew C. McCallum and others, provides multi-perspective annotations for musical structure. By acknowledging that different listeners might place a "chorus" boundary at slightly different times, the dataset allows models to learn the distribution of human perception rather than a single arbitrary point.

In the realm of audio captioning, Yiming Zhang 0025 and colleagues proposed Caption Feature Space Regularization. Since different people use vastly different vocabulary to describe the same sound, they use contrastive learning to align diverse captions into a shared "proxy feature space." This allows the model to understand that "a car driving by" and "motor vehicle passing" are semantically equivalent, even if the words differ.

Aligning with Human Preference

The ultimate goal of this research is to move from "correctness" to "alignment." In MusicRL, Geoffrey Cideron and his team at Google DeepMind applied Reinforcement Learning from Human Feedback (RLHF) to music generation. They recognized that a prompt like "happy music" is highly subjective. By training on 300,000 real-world user preferences, the model learns to generate audio that satisfies the average human expectation, acknowledging that no single objective function can capture the "truth" of a subjective prompt.

By treating ambiguity as a feature of the data rather than a flaw, this research direction is building audio systems that are more robust, more human-centric, and ultimately more capable of navigating the messy, subjective world of sound.

Papers in this version

Showing 60 of 125 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 151 papers
  2. Version 1 · 2026-10-04 · 125 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_perceptual_subjectivity_b0f34500,
  title = {perceptual subjectivity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/perceptual-subjectivity/rdr_b0f34500135643c1ae241473afb842a0},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_modeling_subjective_interpretations_of_s_b0f34500_v1,
  title = {Modeling Subjective Interpretations of Sound and Music},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/modeling-subjective-interpretations-of-sound-and-music/rdr_b0f34500135643c1ae241473afb842a0/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}