TL;DR
Spatial Transfusion is a unified framework that bridges the gap between understanding, generating, and editing spatial audio in First-Order Ambisonics (FOA). By combining a frozen large language model (LLM) backbone with modality-based LoRA routing and a flow-matching objective, the model enables fine-grained, controllable manipulation of complex 3D acoustic scenes through both structured JSON and natural language interfaces.
What Problem Does This Paper Address?
While Large Audio-Language Models (LALMs) have become adept at describing spatial audio scenes (e.g., "a piano playing 2 meters to the left in a large hall"), the tasks of generating those scenes from descriptions and editing existing ones remain largely siloed.
Spatial audio is inherently more complex than mono audio; it involves 3D source locations, distances, room acoustics, and diffuse backgrounds. Existing generative models often lack per-source control in multi-source environments, and editing tools are frequently limited to simple operations like left-right panning. The authors address the need for a unified representation where understanding, generation, and editing are treated as different interfaces to the same underlying spatial scene knowledge.
Methods
The authors introduce Spatial Transfusion, building upon the Sci-Phi architecture. The system utilizes a shared Transformer decoder to handle three distinct tasks:
- Understanding: Autoregressive next-token prediction (Cross-Entropy loss) to generate text descriptions from audio input.
- Generation: Flow-matching (MSE loss) over continuous VAE latents to synthesize FOA audio from text descriptions.
- Editing: A structured transformation where the model receives a source scene and an "atomic edit" instruction (e.g., "move source," "change level") to produce a modified target scene.
Key Technical Innovations:
- Modality-Based LoRA Routing: Instead of full fine-tuning, which the authors found suboptimal, they use separate Low-Rank Adaptation (LoRA) paths for audio and text tokens. This allows the model to learn new generative capabilities without "forgetting" its pretrained understanding.
- Structured Editing Taxonomy: The authors define 12 atomic operations spanning semantic, spatial, acoustic, and temporal dimensions for both directional sources and diffuse backgrounds.
- Training Curriculum: A four-stage progression starting from single-source scenes, moving to editing, then multi-source complexity, and finally "instruction tuning" to translate structured JSON capabilities into natural language.
Datasets
- Training Data: 4.6M unique simulated scenes for understanding/generation and 2.4M for editing. These were created by convolving sound events with simulated Room Impulse Responses (RIRs).
- Evaluation: A held-out test set based on the FOA-MEIR dataset, which uses real-world RIRs and ambient noise to ensure the model generalizes beyond simple simulations.
- Instruction Tuning: Natural language captions were generated using GPT-5.4-mini, and question-answering pairs were sourced from Sci-Phi-Chat.
Key Insights and Ideas
- Understanding as a Foundation: The core hypothesis is that a model must first build a structured, fine-grained understanding of acoustic attributes (position, activity, semantics) to effectively generate or manipulate them.
- The Power of Routing: Modality-based routing (separating audio and text LoRAs) proved superior to "Dense LoRA" (one adapter for all) or "Task-based routing." It provides the most balanced performance across all three tasks.
- Preservation via Concatenation: For editing, simply providing a text instruction isn't enough to preserve the original audio's identity. The authors found that concatenating the clean source latents channel-wise with the noised target latents is essential for high-fidelity editing.
- JSON to Natural Language: Learning a structured JSON interface first provides a "scaffold" for the model. This makes it easier to later map these explicit controls to free-form natural language instructions.
Key Results
- Preserved Understanding: Spatial Transfusion maintains the spatial reasoning capabilities of the base Sci-Phi model, achieving a Tuple Score of 0.812 (see Table 3).
- Generative Superiority: It significantly outperforms full fine-tuning in generation quality, reducing the FAD-PSELD (a spatial-semantic fidelity metric) from 3.82 to 0.964.
- Effective Editing: The model successfully applies edits while preserving the input scene, achieving an EditFLAM score of 17.8, outperforming the full fine-tuning baseline (15.5).
- Natural Language Generalization: After instruction tuning, the model could answer spatial questions with 47.6% accuracy, more than doubling the performance of a cascaded Sci-Phi + GPT baseline (20.8%).
Limitations and Future Work
- Fidelity Gap: While impressive, a performance gap still exists between the model's generation and the "topline" FOA-VAE reconstruction, particularly in audio fidelity (FAD-CLAP).
- Simulated vs. Real: The training relies heavily on simulated data. While the FOA-MEIR test set uses real RIRs, the authors acknowledge that extending the model to handle complex, non-simulated real-world recordings is a key next step.
- Instruction Tuning Trade-offs: The authors noted some loss in editing fidelity when moving from structured JSON to natural language, suggesting room for improvement in how the model balances explicit control with linguistic flexibility.
Summary updated