Unified Architectures for General Audio Intelligence

This direction is split into other directions: cross-domain audio generalization. Its last version stays readable and citable here.

Version 1 of 1523 papers623 conceptsfrozen 2026-10-04

Historically, the field of audio research has been a collection of specialized silos. If you wanted to transcribe speech, you built an Automatic Speech Recognition (ASR) model; if you wanted to compose a melody, you built a Text-to-Music (TTM) system. These domains—speech, music, and environmental sounds—were treated as fundamentally different signals requiring distinct architectures.

However, a new wave of research is dismantling these silos. The core intuition is that all auditory signals share a common acoustic "DNA." Whether it is the prosody of a human voice, the timbre of a cello, or the reverberation of a slamming door, the underlying physics of sound remains consistent. By unifying these domains into single, "foundation" models, researchers are discovering that models can learn cross-domain synergies that specialized systems miss.

The Universal Tokenization Problem

The first hurdle in unification is representation. Traditional codecs were often optimized for speech, discarding the high-frequency nuances essential for music. To bridge this, researchers like Dongchao Yang and Chao Weng introduced HiFi-Codec, which uses Group-Residual Vector Quantization (GRVQ) to compress any audio into a small number of discrete tokens without losing fidelity.

The goal is a "universal language" for audio. UniCodec, developed by Yidi Jiang and colleagues, takes this further by using a domain-adaptive codebook. This allows a single model to represent the vastly different statistical properties of speech and music simultaneously, preventing the "domain interference" that typically occurs when a model tries to learn both at once.

From Task-Specific Models to Audio LLMs

Once audio is tokenized, it can be treated like text. This has led to the rise of Large Audio Language Models (LALMs). Instead of building a model for a specific task, researchers like Xu Tan have proposed UniAudio, a 1B-parameter model that treats all audio tasks—from Text-to-Speech (TTS) to music editing—as a unified sequence-to-sequence problem.

This approach works because it leverages the "foundation model" scaling laws. When a model like Audio Flamingo 3 is trained on massive, diverse datasets, it develops a "flexible thinking" mechanism. It doesn't just recognize patterns; it reasons about them. For instance, Qwen-Audio by Jin Xu can perceive the full spectrum of acoustic signals, allowing it to answer complex questions about a recording that contains both dialogue and background music.

Overcoming Task Conflict with MoE

A major challenge in unification is "task conflict": the features needed to generate clear speech are often at odds with those needed for complex orchestral music. To solve this, Zhenyu Liu and Yunxin Li introduced UniMoE-Audio.

By using a Dynamic-Capacity Mixture-of-Experts (MoE) framework, the model can route different types of audio through specialized "expert" pathways within the same architecture. This allows the model to maintain the high-fidelity requirements of music synthesis without degrading its performance on speech tasks, effectively having its cake and eating it too.

Natural Language as the Universal Interface

The ultimate goal of this research is to provide a single, intuitive interface for all audio interactions. Models like InstructAudio and Audiobox (developed by Apoorv Vyas, Andros Tjandra, and others) allow users to generate or manipulate audio using natural language prompts.

Instead of needing a specialized tool for "voice conversion" and another for "sound effect generation," a user can simply type: "A woman speaking in a whisper over the sound of a rainy forest." Because the model has been trained on a unified representation of both speech and environment, it understands how these elements should interact—how the rain should muffle the voice and how the voice's reverb should match the forest's acoustics.

Unified Evaluation and Perception

Finally, if we have unified models, we need unified ways to judge them. Traditional metrics like Mean Opinion Score (MOS) were often domain-specific. Andros Tjandra and the Meta team addressed this with Audiobox-Aesthetics, a model that assesses quality across speech, music, and sound effects using four universal axes: Production Quality, Complexity, Enjoyment, and Usefulness.

This shift toward unification—from SAM Audio for universal source separation to Audio-FLAN for instruction tuning—suggests that the future of audio AI is not a collection of tools, but a single, cohesive "Audio GPT" that can listen, speak, and compose with equal fluency.

Papers in this version

Showing 60 of 523 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_unified_architectures_for_general_audio__adf61db9,
  title = {Unified Architectures for General Audio Intelligence},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/unified-architectures-for-general-audio-intelligence/rdr_adf61db9191d42a6a31bcee62f4ac46b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_unified_architectures_for_general_audio__adf61db9_v1,
  title = {Unified Architectures for General Audio Intelligence},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/unified-architectures-for-general-audio-intelligence/rdr_adf61db9191d42a6a31bcee62f4ac46b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}