TL;DR
This paper introduces an automated system for mixing multitrack audio that aims to reduce perceptual masking, a common issue that degrades audio clarity. By adapting the MPEG psychoacoustic model and employing optimization techniques, the system can automatically adjust equalization and dynamic range compression to improve perceived audio quality. The study also investigates the benefit of using "subgroups" (grouping similar instruments) in this automated process, finding it can further enhance mix quality.
What Problem Does This Paper Address?
Mixing multitrack audio, the process of balancing and combining individual recorded tracks into a final stereo mix, is a complex and often iterative task. A significant challenge in this process is "auditory masking," where the presence of a loud sound makes it difficult or impossible to hear a quieter sound, leading to a loss of clarity and a "muddy" mix. This masking can occur in both the frequency domain (one sound obscuring another at a similar frequency) and the time domain (a sound obscuring another that occurs shortly before or after it). Traditionally, reducing masking relies on the subjective judgment and experience of a human audio engineer, involving trial-and-error adjustments of levels, equalization (EQ), and dynamics processing. This paper aims to automate this masking minimization process, making it more efficient and potentially accessible to a wider range of users.
Methods
The core of the proposed system lies in an automated masking minimization framework. This framework utilizes:
- A Novel Multitrack Masking Metric: Inspired by the MPEG psychoacoustic model, this metric quantifies the amount of masking present between different tracks in a multitrack recording. It adapts the MPEG model's approach to spectral analysis and masking threshold calculation to work across multiple audio sources simultaneously.
- Audio Processing Techniques: The system employs two key audio processing techniques:
- Equalization (EQ): A six-band equalizer is used to adjust the frequency content of individual tracks.
- Dynamic Range Compression (DRC): This process controls the loudness variations within a track.
- Numerical Optimization: A Particle Swarm Optimization (PSO) algorithm is used to iteratively adjust the parameters of the EQ and DRC to minimize the calculated masking metric. The system defines objective functions that aim to reduce overall masking and balance masking levels across tracks.
- Subgrouping: The researchers investigate the impact of grouping similar instrument tracks (e.g., all drums, all vocals) into "subgroups." This allows for processing to be applied to these groups, potentially leading to more efficient masking reduction and improved perceived quality.
Datasets
The study used five songs from the "Open Multitrack Testbed" [65] for their experiments. These songs varied in genre and number of tracks. The paper does not introduce new datasets but utilizes existing multitrack recordings for evaluation.
Key Insights and Ideas
- Automating Subjectivity: The paper tackles the inherently subjective process of audio mixing by developing an objective metric for masking and using optimization to guide the process.
- MPEG Model Adaptation: Leveraging the well-established MPEG psychoacoustic model as a foundation for a multitrack masking metric is a clever way to ground the system in established perceptual principles.
- Subgrouping as a Shortcut: The hypothesis that grouping similar instruments (subgrouping) can lead to more efficient masking reduction is intuitively appealing, as these instruments often share similar spectral and dynamic characteristics.
- Beyond Simple Leveling: The approach goes beyond just adjusting track levels, incorporating EQ and dynamics processing, which are crucial for fine-grained masking control.
Key Results
The study found that:
- Reduced Masking: The automated system successfully reduced inter-channel auditory masking in multitrack mixes. Objective measurements showed a significant decrease in masking after optimization.
- Improved Perceived Quality: Subjective listening tests indicated that the automated mixes were perceived as having better quality and clarity compared to a simple summed mix (all tracks combined without processing).
- Benefit of Subgrouping: The use of subgroups in the automatic mixing process generally led to improved perceived quality and clarity. This was supported by both objective masking reduction (fewer iterations needed) and subjective preference ratings.
- Human Mixes Still Lead: While the automated mixes performed well, they did not consistently outperform human-engineered mixes, particularly in terms of overall preference. However, in some specific cases (e.g., "Lead Me" for clarity), the automated mix with subgroups was rated comparably or even better than a low-quality human mix.
- Limited Impact on Emotion: The study found little evidence that the automatic mixing process, with or without subgroups, significantly impacted the perceived emotion of the music.
Limitations and Future Work
- Static Processing: A limitation acknowledged is that the EQ and DRC settings are static for the entire track. Future work could involve dynamic processing that adapts these parameters on a frame-by-frame basis according to the masking metric.
- Human Mix Performance: The automated system did not consistently surpass human mixes, suggesting room for improvement in capturing the nuances of professional mixing.
- Perceived Emotion: The lack of significant impact on perceived emotion suggests that while masking reduction is important for clarity, it may not be the primary driver of emotional perception in music.
- Panning: The authors suggest incorporating panning (stereo placement) into the optimization process as a future direction.
- Larger Optimality Tolerance: Exploring a larger optimality tolerance in the optimization process, potentially followed by a different solver, could lead to more efficient solutions.
Summary updated