Spatial Transfusion: From Structured Spatial Scene Understanding to Generation and Editing

Jaeyeon Kim, Sebastian Braun, Hannes GamperarXiv 2026 · arXiv

TL;DR

Spatial Transfusion is a unified framework that bridges the gap between understanding, generating, and editing spatial audio in First-Order Ambisonics (FOA). By combining a frozen large language model (LLM) backbone with modality-based LoRA routing and a flow-matching objective, the model enables fine-grained, controllable manipulation of complex 3D acoustic scenes through both structured JSON and natural language interfaces.

What Problem Does This Paper Address?

While Large Audio-Language Models (LALMs) have become adept at describing spatial audio scenes (e.g., "a piano playing 2 meters to the left in a large hall"), the tasks of generating those scenes from descriptions and editing existing ones remain largely siloed.

Spatial audio is inherently more complex than mono audio; it involves 3D source locations, distances, room acoustics, and diffuse backgrounds. Existing generative models often lack per-source control in multi-source environments, and editing tools are frequently limited to simple operations like left-right panning. The authors address the need for a unified representation where understanding, generation, and editing are treated as different interfaces to the same underlying spatial scene knowledge.

Methods

The authors introduce Spatial Transfusion, building upon the Sci-Phi architecture. The system utilizes a shared Transformer decoder to handle three distinct tasks:

  1. Understanding: Autoregressive next-token prediction (Cross-Entropy loss) to generate text descriptions from audio input.
  2. Generation: Flow-matching (MSE loss) over continuous VAE latents to synthesize FOA audio from text descriptions.
  3. Editing: A structured transformation where the model receives a source scene and an "atomic edit" instruction (e.g., "move source," "change level") to produce a modified target scene.
Fig. 1. Overview of Spatial Transfusion. A shared Transformer backbone serves (a) understanding with a cross-entropy loss on text, and (b) generation and (c) editing with a flow-matching loss on target FOA-VAE latents. In editing, clean source latents are additionally concatenated channel-wise with the noised target latents.

Key Technical Innovations:

  • Modality-Based LoRA Routing: Instead of full fine-tuning, which the authors found suboptimal, they use separate Low-Rank Adaptation (LoRA) paths for audio and text tokens. This allows the model to learn new generative capabilities without "forgetting" its pretrained understanding.
  • Structured Editing Taxonomy: The authors define 12 atomic operations spanning semantic, spatial, acoustic, and temporal dimensions for both directional sources and diffuse backgrounds.
  • Training Curriculum: A four-stage progression starting from single-source scenes, moving to editing, then multi-source complexity, and finally "instruction tuning" to translate structured JSON capabilities into natural language.

Datasets

  • Training Data: 4.6M unique simulated scenes for understanding/generation and 2.4M for editing. These were created by convolving sound events with simulated Room Impulse Responses (RIRs).
  • Evaluation: A held-out test set based on the FOA-MEIR dataset, which uses real-world RIRs and ambient noise to ensure the model generalizes beyond simple simulations.
  • Instruction Tuning: Natural language captions were generated using GPT-5.4-mini, and question-answering pairs were sourced from Sci-Phi-Chat.

Key Insights and Ideas

  • Understanding as a Foundation: The core hypothesis is that a model must first build a structured, fine-grained understanding of acoustic attributes (position, activity, semantics) to effectively generate or manipulate them.
  • The Power of Routing: Modality-based routing (separating audio and text LoRAs) proved superior to "Dense LoRA" (one adapter for all) or "Task-based routing." It provides the most balanced performance across all three tasks.
  • Preservation via Concatenation: For editing, simply providing a text instruction isn't enough to preserve the original audio's identity. The authors found that concatenating the clean source latents channel-wise with the noised target latents is essential for high-fidelity editing.
  • JSON to Natural Language: Learning a structured JSON interface first provides a "scaffold" for the model. This makes it easier to later map these explicit controls to free-form natural language instructions.
Fig. 2. Structured representations used in Stages 1–3. (a) A scene description lists directional sources with their attributes plus room and background fields; it is the understanding output and the gener- ation input. (b) An edit instruction names an operation from Table 1, its target, and the modification to apply; it is the editing input.

Key Results

  • Preserved Understanding: Spatial Transfusion maintains the spatial reasoning capabilities of the base Sci-Phi model, achieving a Tuple Score of 0.812 (see Table 3).
  • Generative Superiority: It significantly outperforms full fine-tuning in generation quality, reducing the FAD-PSELD (a spatial-semantic fidelity metric) from 3.82 to 0.964.
  • Effective Editing: The model successfully applies edits while preserving the input scene, achieving an EditFLAM score of 17.8, outperforming the full fine-tuning baseline (15.5).
  • Natural Language Generalization: After instruction tuning, the model could answer spatial questions with 47.6% accuracy, more than doubling the performance of a cascaded Sci-Phi + GPT baseline (20.8%).
Table 3. Understanding results on the FOA-MEIR-based test set.

Limitations and Future Work

  • Fidelity Gap: While impressive, a performance gap still exists between the model's generation and the "topline" FOA-VAE reconstruction, particularly in audio fidelity (FAD-CLAP).
  • Simulated vs. Real: The training relies heavily on simulated data. While the FOA-MEIR test set uses real RIRs, the authors acknowledge that extending the model to handle complex, non-simulated real-world recordings is a key next step.
  • Instruction Tuning Trade-offs: The authors noted some loss in editing fidelity when moving from structured JSON to natural language, suggesting room for improvement in how the model balances explicit control with linguistic flexibility.

Summary updated

Abstract

Spatial audio conveys not only what sounds are present, but where they originate and how they interact with the environment. Recent audio-language models increasingly understand and reason about such scenes, yet spatial audio understanding, generation, and editing remain largely separate. We present Spatial Transfusion, a unified first-order ambisonics (FOA) framework that extends a pretrained spatial audio understanding model to generation and editing through flow matching over VAE latents. We formulate editing as a structured scene transformation over a taxonomy of atomic edits. Keeping the pretrained backbone frozen, we introduce modality-based LoRA routing and a curriculum that progressively adds generation, editing, and natural-language interaction. Experiments show that Spatial Transfusion largely preserves spatial understanding while outperforming full fine-tuning in generation and editing. The resulting model supports controllable FOA generation and editing through both structured and natural-language interfaces. Audio samples are available at https://microsoft.github.io/spatial-transfusion/
BibTeX
@misc{kim2026spatial,
  author = {Jaeyeon Kim and Sebastian Braun and Hannes Gamper},
  title = {{Spatial Transfusion: From Structured Spatial Scene Understanding to Generation and Editing}},
  year = {2026},
  eprint = {2611.02330},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.02330},
}
Figure