TL;DR
MARIMBA is a unified "System One" model for speech that replaces free-form text generation with fast, calibrated "typed" decisions (e.g., yes/no, multiple choice). Built on a frozen 0.6B Qwen3-ASR encoder, it matches or outperforms 50x larger models like Qwen3-Omni on tasks like term spotting, turn-end detection, and speaker counting while maintaining low latency and high reliability.
What Problem Does This Paper Address?
Modern speech applications often juggle multiple distinct tasks simultaneously: detecting when a user finishes speaking (turn-end detection), spotting specific keywords (term spotting), identifying the language, and counting speakers.
Currently, researchers typically choose between two extremes:
- Single-task tools: Highly specialized models (like VAD or keyword spotters) that lack a shared interface and require separate feature extraction.
- Audio Language Models (LLMs): Large models that "reason" through text generation. While flexible, they are computationally expensive, slow, and their outputs are notoriously difficult to calibrate—meaning you can't easily trust the "probability" they assign to an answer.
MARIMBA addresses this by providing a "System One" interface—referring to the fast, automatic, and intuitive processing layer of the brain. It offers a shared, request-time interface for diverse speech decisions without the overhead of full autoregressive text generation for every task.
Methods
The core of MARIMBA is a frozen Qwen3-ASR 0.6B encoder. By keeping the encoder frozen, the authors ensure that features are extracted only once per audio window, regardless of how many questions are asked.
The system routes questions through three specialized paths:
- Lexical Path: For term spotting, the model uses the frozen ASR decoder to generate a greedy transcript plus four beam-search hypotheses. It checks for exact matches and applies Platt scaling to turn match counts into calibrated probabilities.
- Non-Lexical Path: Tasks like voice activity or speaker change use lightweight "readout" heads—simple logistic regressions or small MLPs—trained on intermediate layers (e.g., layer 12 for speakers, layer 18 for turn-ends).
- Question-Conditioned Head: A cross-attention mechanism allows the model to answer arbitrary categorical questions by attending to the audio features based on the text of the question and its options.
Crucially, MARIMBA outputs typed JSON (choice, yes/no, or score) rather than free text, ensuring the output space is constrained and the uncertainties are mathematically sound.
Datasets
The authors evaluated MARIMBA across a massive battery of public and private data:
- Public Benchmarks: Common Voice, FLEURS (multilingual), Speech Commands, AMI (meetings), VoxConverse (diarization), and MUSAN (noise).
- Turn-End Specific: Data from Smart Turn v3.2 and ICSI meeting headset mixes.
- Private Stress Tests: A unique dataset of Brazilian Portuguese medical consultations. This is particularly challenging due to rare medication names and "phonetic near-misses" (words that sound similar but have different meanings).
- Synthetic PII: 200 synthesized clips used to test the detection of personal information like taxpayer IDs and phone numbers.
Key Insights and Ideas
- Frozen Encoders are Sufficient: One of the most surprising findings is that a frozen ASR encoder already contains almost all the information needed for paralinguistic tasks (like speaker counting) and structural tasks (like turn detection). You don't need to fine-tune the whole stack; you just need the right "probes" on the right layers.
- Calibration is King: In production, knowing when a model is unsure is as important as being right. By using "typed" decisions, MARIMBA can use standard calibration techniques (like Expected Calibration Error or ECE) that are impossible with free-form text LLMs.
- The Power of Beam Search for Spotting: Instead of relying on a single "best" transcript, MARIMBA uses the top 5 hypotheses from the ASR decoder. This allows it to recover terms that might have been "lost" in the primary decoding path, significantly boosting AUROC for rare terms.
Key Results
MARIMBA demonstrates that "smaller and specialized" can beat "massive and general":
- Vs. Audio LLMs: On 6 out of 9 public tests, the 0.6B MARIMBA matched or beat the 30B Qwen3-Omni. For example, on FLEURS term detection, MARIMBA achieved an AUROC of 0.901 compared to Qwen3-Omni's 0.791.
- Speaker Counting: It outperformed the popular
pyannote 3.1library in speaker counting accuracy on the AMI and VoxConverse datasets. - Turn-End Detection: It reached an AUROC of 0.930, coming close to the performance of specialized, state-of-the-art models like Smart Turn v3.2 (0.982).
- Latency: While lexical tasks (requiring the decoder) take ~2.35s, structural tasks like turn-end detection and voice activity are lightning-fast, with p95 latencies of just 13–16ms.
Limitations and Future Work
Despite its strengths, MARIMBA has clear boundaries:
- ASR Dependency: Lexical decisions are only as good as the underlying ASR. If the Qwen3-ASR model consistently mishears a specific rare drug name, MARIMBA cannot "spot" it.
- Speaker Undercounting: While accurate for 0, 1, or 2 speakers, the model tends to undercount when 3 or more people are speaking simultaneously.
- Task Limits: The model failed to generalize to "unseen" tasks like scene classification (e.g., identifying if audio is from a "street" vs. "office"), staying at chance levels. This suggests the frozen ASR features are heavily biased toward speech and struggle with general environmental audio.
Future work may involve exploring "System Two" integration, where MARIMBA acts as a fast filter that triggers a larger, more capable LLM only when complex reasoning is required.
Summary updated