TAC2: Timestamped Audio Captioning as composable functions

Swapnil Bhosale, Prem Seetharaman, Ke Chen, Haonan Chen, Justin Salamon, Zeyu JinarXiv 2026 · arXiv

TL;DR

TAC2 is a new foundation model that transforms audio captioning into a series of modular, composable functions, producing granular and temporally precise acoustic timelines. By decoupling raw perception from high-level reasoning and "eliciting" hidden paralinguistic knowledge from pre-trained models, it achieves state-of-the-art performance in both complex audio understanding and controllable text-to-audio generation.

What Problem Does This Paper Address?

While modern Large Audio-Language Models (LALMs) are great at having conversations about sounds, they often struggle with precision. Most current models produce free-form, "chatty" descriptions that lack exact timestamps and conflate what the model hears with what it thinks. This makes them difficult to use for practical applications like semantic audio search, precise editing, or automated archiving, which require structured, queryable data grounded in physical time.

The authors argue that human hearing naturally separates overlapping signals before reasoning about them. TAC2 aims to replicate this by creating a structured "acoustic timeline" that explicitly identifies who spoke, what they said, and exactly when every event—from a door slam to a specific word emphasis—occurred.

Methods

The TAC2 framework is built in two main stages, centered around a novel Instruction-Input-Output (IIO) format.

Stage 1: Building the Perceptual Foundation

To avoid the impossible cost of manually labeling millions of audio clips with precise timestamps, the authors used Multi-view Perceptual Fusion. They took off-the-shelf "weak teachers" (specialized models for sound detection, speaker diarization, and captioning) and programmatically fused their predictions. By aligning these different "views" on a shared time axis using Intersection-over-Union (IoU) matching, they created high-fidelity training targets from massive amounts of unlabeled "in-the-wild" audio.

The two-stage TAC2 framework overview, showing data preparation and speech elicitation.

The IIO Completion Format

Instead of a standard one-shot captioner, TAC2 uses an Instruction-Input-Output (IIO) format. This treats every task (like transcribing, identifying speakers, or detecting emotions) as a composable function.

  • Instruction: Defines the task (e.g., "diarize").
  • Input: Optional context, like a pre-existing transcript.
  • Output: The requested structured data.

This modularity allows the model to work in a single pass or as a "cascade" where earlier outputs (or external data like a human-verified transcript) condition later ones.

The IIO completion format allows for modular, multi-pass inference by feeding preceding outputs back as inputs.

Stage 2: Latent Capability Elicitation (TAC2-S)

The authors discovered through "probing" that pre-trained audio models already "know" a lot about paralinguistics (like stress, emotion, and accent) in their internal layers, even if they don't output that information. They trained lightweight "specialists" on tiny gold-standard datasets (<10k samples) to "surface" these latent traits. These specialists were then distilled into a unified model called TAC2-S.

Datasets

  • Training: Approximately 2 million licensed, in-the-wild audio clips (up to 30s each).
  • Specialist Elicitation: Small "gold" corpora including IEMOCAP, RAVDESS, and MELD (for emotion), Common Voice (for accents), and A560E (for word stress).
  • Evaluation: A battery of benchmarks including MMSU (spoken understanding), MMAR/MMAU (general audio), MSU-Bench (multi-speaker scenarios), and LongAudioSpan (for long-form audio up to 2.7 hours).

Key Insights and Ideas

  • Decoupling Perception from Reasoning: One of the biggest "aha" moments was finding that a structured schema generated by TAC2, when read by a frozen text-only LLM, often outperforms end-to-end audio models. This suggests that the bottleneck in current LALMs isn't reasoning, but the fidelity of their initial perception.
  • Latent Perception: The model doesn't need to be taught how to "hear" emotion from scratch; it already does. It just needs a small amount of tuning to learn how to describe that internal state in a structured way.
  • Compositionality as a Feature: The IIO format means TAC2 can act as a "plug-and-play" component. If you already have a high-quality transcript from another system, you can feed it into TAC2 to get perfectly aligned speaker turns and emotional markers without the model trying to re-transcribe the audio.

Key Results

  • Superior Understanding: TAC2-S achieved an 81.9 overall score on MMSU, outperforming major baselines like Gemini-3.7 Flash and Qwen3Omni-Captioner.
  • Speaker-Centric Accuracy: On the challenging MSU-Bench, it reached 82.2 overall, showing significantly more robust perception of speaker identity and traits than native multimodal models.
  • High-Fidelity Generation: When used as a "control interface" for text-to-audio generation, TAC2-S enabled precise zero-shot control over event timing and word-level emphasis. It achieved a temporal grounding error of just 1.54s, nearly a full second better than previous timestamp-conditioned models.
Perception fidelity comparison: TAC2-S (blue) shows superior performance across various acoustic and speaker attributes compared to baselines.

Limitations and Future Work

The authors note that the current schema prioritizes speech attributes over non-speech sounds and music. Additionally, the model sometimes merges turns if two consecutive voices sound very similar, leading to an under-count of unique speakers.

Future work aims to move beyond human-designed schemas. The authors envision letting descriptions evolve autonomously via reinforcement learning, where a text-only reasoner "rewards" the model for extracting the specific cues needed to solve a given task.

Summary updated

Abstract

Real-world audio applications require precise, queryable representations grounded in physical time. Yet modern Large Audio-Language Models (LALMs) increasingly favor conversational generation, producing free-form descriptions that conflate acoustic perception with higher-level reasoning and lack precise temporal grounding. We introduce TAC2, a foundation model that casts timestamped audio captioning as composable functions to construct granular, speaker-attributed acoustic timelines directly from in-the-wild audio. To avoid prohibitively expensive manual annotation, we programmatically fuse complementary predictions from specialized weak teachers through time-registered IoU matching. We model the resulting multi-granularity schema of timestamped events, speaker turns, and fine-grained speech attributes using an Instruction-Input-Output (IIO) completion format, in which each task is a function of the audio, an instruction, and an optional input. A single call decodes the full description autonomously, while composing calls lets earlier outputs or external priors, such as a reference transcript, condition later ones. We further extend TAC2 with expressive speech attributes to obtain TAC2-S, using parameter-efficient specialists to elicit latent paralinguistic capabilities from the pre-trained backbone and distill them into a unified model. We validate the resulting representation in two complementary settings. Paired with frozen text-only LLMs, it yields strong spoken audio understanding across diverse reasoning backends, demonstrating the utility of decoupling fine-grained acoustic perception from downstream reasoning. Used as paired supervision for text-to-audio generation, it supports semantically and temporally grounded reconstruction, together with explicit zero-shot control over event timing, speaker attributes, and word-level emphasis.
BibTeX
@misc{bhosale2026tac,
  author = {Swapnil Bhosale and Prem Seetharaman and Ke Chen and Haonan Chen and Justin Salamon and Zeyu Jin},
  title = {{TAC2: Timestamped Audio Captioning as composable functions}},
  year = {2026},
  eprint = {2610.36568},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.36568},
}
Figure