TL;DR
AURA demonstrates that general-purpose audio codecs, even those trained without spatial supervision, already capture the "where" of a sound alongside the "what." By introducing a "six-slot" representation that highlights inter-channel differences and a "scene supervision" training strategy, AURA outperforms specialized spatial models on the SpatialSoundQA benchmark by over 10 points while using 60% fewer trainable parameters.
What Problem Does This Paper Address?
Most Large Audio-Language Models (LALMs) are excellent at identifying what a sound is (e.g., a dog barking) but struggle to determine where it is coming from. To solve this, current state-of-the-art systems typically add a second, dedicated "spatial encoder" specifically trained on localized audio.
However, this approach has two major drawbacks: it splits semantic and spatial processing into two different feature spaces, and it requires expensive pre-training on specialized datasets. The authors of AURA ask a fundamental question: Is this dedicated hardware really necessary? Can a single, general-purpose audio encoder handle both identity and location?
Methods
The researchers first conducted a "layer-wise analysis" of X-Codec, a general-purpose audio codec. They discovered that while shallow layers focus on the "what" (semantic content), deeper layers naturally capture the "where" (spatial cues like interaural time and level differences).
To unlock this latent spatial intelligence, they developed AURA, which uses two key innovations:
- Six-Slot Binaural Representation: Instead of just passing raw audio features, AURA organizes them into six "slots" per frame. It separates content-oriented and acoustic-oriented bands for both the left and right ears and—crucially—adds two slots for the explicit difference between the channels. This makes the lateral cues (left vs. right) immediately accessible to the model.
- Complete Alignment (Scene Supervision): The authors identified an "information asymmetry" problem. In a scene with two sounds, the training data usually only asks about one. This makes it hard for the model to "bind" a specific location to a specific sound. AURA is trained to describe the entire scene—identifying and locating every audible source—before answering the specific question.
Datasets
- SpatialSoundQA (SSQA): The primary benchmark for training and evaluation, featuring binaural audio rendered in simulated 3D environments.
- MMAU, MMAR, and AIR-Bench: General-purpose audio benchmarks used to ensure that AURA’s spatial tuning didn't break its ability to understand general sounds.
- MRSSpeech: Real-world binaural recordings used to verify that the spatial information found in the codec wasn't just an artifact of simulation.
Key Insights and Ideas
- The Info is Already There: You don't need a spatial encoder to "see" space; general-purpose codecs already capture it in their deeper residual vector quantization (RVQ) layers.
- The "Binding" Failure: The biggest hurdle in spatial audio isn't "hearing" the direction; it's the "binding problem." In multi-source scenes, models often know there is a sound to the left and a sound to the right, but they struggle to remember which sound is where.
- Supervision is the Secret Sauce: By forcing the model to describe the whole scene (Scene Supervision), the researchers solved the binding problem. This "complete alignment" between audio and text is what allows a frozen, general encoder to beat specialized ones.
Key Results
AURA achieves impressive results while remaining highly efficient:
- Superior Accuracy: On the dual-source direction task of SpatialSoundQA, AURA scored 48.21%, beating the strongest dedicated baseline (DSpAST-BAT at 37.90%) by 10.31 points.
- Efficiency: AURA uses only 64.63M trainable parameters—roughly 40% of the 161M parameters used by its competitors.
- Better Generalization: Unlike dedicated spatial models, which often lose their "general" audio sense, AURA outperformed all spatial baselines on general-audio benchmarks like MMAU (44.50% vs. 34.90% for the next best spatial model).
Limitations and Future Work
While AURA proves that a unified representation is possible, some limitations remain:
- Simulated Training: The model is primarily trained on simulated binaural data. While probing on the MRSSpeech dataset showed promising results, more work is needed to see how well these models perform in complex, real-world acoustic environments.
- Sequence Length: The "six-slot" representation increases the number of tokens the language model has to process, which could lead to higher computational costs for very long audio clips.
- Static Sources: The current evaluation focuses on static sound sources; future work could extend this to moving sources and dynamic listener head-tracking.
Summary updated