TL;DR
TAC2 is a new foundation model that transforms audio captioning into a series of modular, composable functions, producing granular and temporally precise acoustic timelines. By decoupling raw perception from high-level reasoning and "eliciting" hidden paralinguistic knowledge from pre-trained models, it achieves state-of-the-art performance in both complex audio understanding and controllable text-to-audio generation.
What Problem Does This Paper Address?
While modern Large Audio-Language Models (LALMs) are great at having conversations about sounds, they often struggle with precision. Most current models produce free-form, "chatty" descriptions that lack exact timestamps and conflate what the model hears with what it thinks. This makes them difficult to use for practical applications like semantic audio search, precise editing, or automated archiving, which require structured, queryable data grounded in physical time.
The authors argue that human hearing naturally separates overlapping signals before reasoning about them. TAC2 aims to replicate this by creating a structured "acoustic timeline" that explicitly identifies who spoke, what they said, and exactly when every event—from a door slam to a specific word emphasis—occurred.
Methods
The TAC2 framework is built in two main stages, centered around a novel Instruction-Input-Output (IIO) format.
Stage 1: Building the Perceptual Foundation
To avoid the impossible cost of manually labeling millions of audio clips with precise timestamps, the authors used Multi-view Perceptual Fusion. They took off-the-shelf "weak teachers" (specialized models for sound detection, speaker diarization, and captioning) and programmatically fused their predictions. By aligning these different "views" on a shared time axis using Intersection-over-Union (IoU) matching, they created high-fidelity training targets from massive amounts of unlabeled "in-the-wild" audio.
The IIO Completion Format
Instead of a standard one-shot captioner, TAC2 uses an Instruction-Input-Output (IIO) format. This treats every task (like transcribing, identifying speakers, or detecting emotions) as a composable function.
- Instruction: Defines the task (e.g., "diarize").
- Input: Optional context, like a pre-existing transcript.
- Output: The requested structured data.
This modularity allows the model to work in a single pass or as a "cascade" where earlier outputs (or external data like a human-verified transcript) condition later ones.
Stage 2: Latent Capability Elicitation (TAC2-S)
The authors discovered through "probing" that pre-trained audio models already "know" a lot about paralinguistics (like stress, emotion, and accent) in their internal layers, even if they don't output that information. They trained lightweight "specialists" on tiny gold-standard datasets (<10k samples) to "surface" these latent traits. These specialists were then distilled into a unified model called TAC2-S.
Datasets
- Training: Approximately 2 million licensed, in-the-wild audio clips (up to 30s each).
- Specialist Elicitation: Small "gold" corpora including IEMOCAP, RAVDESS, and MELD (for emotion), Common Voice (for accents), and A560E (for word stress).
- Evaluation: A battery of benchmarks including MMSU (spoken understanding), MMAR/MMAU (general audio), MSU-Bench (multi-speaker scenarios), and LongAudioSpan (for long-form audio up to 2.7 hours).
Key Insights and Ideas
- Decoupling Perception from Reasoning: One of the biggest "aha" moments was finding that a structured schema generated by TAC2, when read by a frozen text-only LLM, often outperforms end-to-end audio models. This suggests that the bottleneck in current LALMs isn't reasoning, but the fidelity of their initial perception.
- Latent Perception: The model doesn't need to be taught how to "hear" emotion from scratch; it already does. It just needs a small amount of tuning to learn how to describe that internal state in a structured way.
- Compositionality as a Feature: The IIO format means TAC2 can act as a "plug-and-play" component. If you already have a high-quality transcript from another system, you can feed it into TAC2 to get perfectly aligned speaker turns and emotional markers without the model trying to re-transcribe the audio.
Key Results
- Superior Understanding: TAC2-S achieved an 81.9 overall score on MMSU, outperforming major baselines like Gemini-3.7 Flash and Qwen3Omni-Captioner.
- Speaker-Centric Accuracy: On the challenging MSU-Bench, it reached 82.2 overall, showing significantly more robust perception of speaker identity and traits than native multimodal models.
- High-Fidelity Generation: When used as a "control interface" for text-to-audio generation, TAC2-S enabled precise zero-shot control over event timing and word-level emphasis. It achieved a temporal grounding error of just 1.54s, nearly a full second better than previous timestamp-conditioned models.
Limitations and Future Work
The authors note that the current schema prioritizes speech attributes over non-speech sounds and music. Additionally, the model sometimes merges turns if two consecutive voices sound very similar, leading to an under-count of unique speakers.
Future work aims to move beyond human-designed schemas. The authors envision letting descriptions evolve autonomously via reinforcement learning, where a text-only reasoner "rewards" the model for extracting the specific cues needed to solve a given task.
Summary updated