TL;DR
INRAS (Implicit Neural Representation for Audio Scenes) is a lightweight neural network that learns a continuous, implicit representation of room acoustics. By disentangling static scene geometry from moving emitter and listener positions, it can render high-fidelity, time-domain binaural impulse responses for any location in real-time. It requires less than 3MB of storage and significantly outperforms existing neural and traditional acoustic encoding methods.
What Problem Does This Paper Address?
To create truly immersive virtual environments, we need accurate spatial audio—meaning the sound you hear must accurately reflect how acoustic waves bounce off the specific walls, floors, and objects in a room. Traditionally, this is modeled using an Impulse Response (IR) between an emitter (sound source) and a listener.
However, computing these IRs on the fly using wave-based physics is computationally prohibitive. Geometry-based approximations (like ray tracing) are faster but struggle with low-frequency wave effects like diffraction. The standard workaround is to pre-compute and store IRs for a dense grid of locations, but this requires massive storage (hundreds of megabytes per room) and relies on interpolation for unmapped points.
Recent deep learning approaches, like Neural Acoustic Fields (NAF), attempt to compress this into a neural network by predicting frequency-time spectrograms. Unfortunately, spectrograms discard phase information and tend to smooth over the sharp, sparse peaks of early sound reflections, which are critical for human directional perception and audio clarity.
Methods
INRAS takes inspiration from classical "acoustic radiance transfer" algorithms, which separate the static geometry of a room from the moving sound sources and listeners. The authors design a two-stage neural architecture that predicts the IR directly in the time domain:
- Audio Scene Feature Decomposition: The network places a set of uniform "bounce points" along the physical boundaries of the room. It then uses three parallel modules:
- Scatter Module: Learns features based on the relative distance from the sound emitter to all bounce points.
- Bounce Module: A Multi-Layer Perceptron (MLP) that learns scene-dependent features shared across all positions (representing the static room geometry).
- Gather Module: Learns features based on the relative distance from the listener to all bounce points.
- Spatial Binaural IR Prediction: A Listener Module fuses the outputs of the three modules along with the listener's head orientation. It uses a set of learned time-domain basis functions to efficiently generate the final binaural impulse response.
Because predicting highly oscillatory time-domain audio is notoriously difficult, INRAS is trained using a combination of Mean Square Error (MSE) and a multi-resolution Short-Time Fourier Transform (STFT) loss, which optimizes spectral convergence, magnitude, and phase simultaneously.
Datasets
- Soundspaces: A simulated dataset containing dense pairs of impulse responses generated by geometric sound propagation. The authors used 6 different scenes (multi-room layouts, non-rectangular rooms, and standard rectangular rooms).
- MeshRIR (S1-M3969 subset): A real-world dataset containing dense, actual impulse response recordings captured in a physical room, used to prove the model works on real data, not just simulations.
Key Insights and Ideas
- Disentanglement is highly efficient: By forcing the network to separate the static room geometry (Bounce Module) from the moving coordinates (Scatter/Gather Modules), the model learns highly reusable features. This drastically shrinks the parameter count.
- Multi-scene scaling is practically free: Because of this disentangled architecture, INRAS can be trained on multiple different rooms simultaneously. You simply add the bounce points for the new rooms and mask out the irrelevant ones during training. This adds only a handful of parameters (~0.1M) to support multiple scenes, whereas traditional storage scales linearly.
- Time-domain > Spectrograms for IRs: Directly predicting the time-domain waveform (guided by STFT loss) allows the model to accurately capture the sharp, high-amplitude peaks of early reflections. Spectrogram-based models tend to blur these out, degrading the perceived clarity of the audio.
Key Results
INRAS achieves state-of-the-art performance across quality, speed, and storage:
- Acoustic Quality: Compared to the previous state-of-the-art (NAF), INRAS improved the C50 error (a metric for early reflection clarity) by 43% (0.6 dB vs 1.06 dB) and the Early Decay Time (EDT) error by 39% (0.019s vs 0.031s).
- Storage: The INRAS model requires only 0.67 million parameters, taking up just 2.56 MB of space. For context, NAF takes 8.55 MB, and traditional AAC/Opus encoding of the same acoustic field takes 180–340 MB.
- Speed: INRAS renders an impulse response in 9.47 ms on a Tesla T4 GPU, making it roughly 4x faster than NAF (37.86 ms) and well within the budget for real-time interactive applications.
- Real-World Superiority: On the real-world MeshRIR dataset, INRAS actually outperformed traditional geometry-based simulations, reducing T60 (reverberation time) error from 1.45% to 0.35%.
Limitations and Future Work
While the authors defer their full limitations discussion to the supplementary material, a few constraints are apparent from the methodology. First, INRAS relies on sampling "bounce points" along the boundaries of the scene, which implies the exact 3D geometry/mesh of the room must be known beforehand. Second, the current formulation assumes static room geometry; it does not account for dynamic environments where large objects (like doors opening or furniture moving) alter the acoustic field in real-time. Future work will likely explore scaling this to even larger, more complex real-world datasets and potentially adapting the implicit representation to handle dynamic, changing scenes.
Summary updated