NIIRF: Neural IIR Filter Field for HRTF Upsampling and Personalization

Yoshiki Masuyama, Gordon Wichern, François G. Germain, Zexu Pan, Sameer Khurana, Chiori Hori, Jonathan Le Roux

TL;DR

This paper introduces NIIRF, a novel neural network approach that models Head-Related Transfer Functions (HRTFs) using cascaded Infinite Impulse Response (IIR) filters. By learning IIR filter parameters directly, NIIRF achieves comparable or superior HRTF upsampling accuracy compared to existing methods, especially when training data is sparse, and offers a more efficient representation than FIR filters.

What Problem Does This Paper Address?

Head-Related Transfer Functions (HRTFs) are crucial for creating realistic 3D audio experiences, as they describe how sound waves are modified by a person's head, torso, and ears. Measuring HRTFs for all possible sound source directions is time-consuming and labor-intensive. Therefore, techniques for spatial upsampling (estimating HRTFs for unmeasured directions) and personalization (adapting generic HRTFs to individual users) are essential. While neural fields (NFs) have shown promise in upsampling by learning to map sound source directions to HRTF representations, existing methods often focus on estimating the magnitude response and converting it to Finite Impulse Response (FIR) filters. FIR filters can be computationally expensive and require many coefficients to accurately represent HRTFs. This paper addresses the need for a more efficient and accurate method for HRTF upsampling and personalization that can leverage the inherent modal nature of HRTFs.

Methods

The core innovation is the Neural IIR Filter Field (NIIRF). Instead of learning HRTF magnitude responses or FIR filter coefficients, NIIRF learns the parameters of a cascade of Infinite Impulse Response (IIR) filters. This cascade typically consists of a low-shelf filter, several peaking filters, and a high-shelf filter, as illustrated in Figure 2. The parameters for these filters (e.g., center frequency, bandwidth, gain) are directly predicted by a neural network (specifically, a Multi-Layer Perceptron with Random Fourier Features for input conditioning) based on the sound source direction (azimuth and elevation).

The training process leverages Differentiable Digital Signal Processing (DDSP). This allows the entire IIR filter cascade to be differentiable, enabling the use of backpropagation to optimize the neural network's parameters. The loss function used is a Mean Squared Error (MSE) on the log-spectral distortion, similar to previous NF-based methods, but applied to the output of the IIR filter cascade.

For personalization, the authors explore several techniques:

  • Conditioning by Concatenation (CbC) and Feature-wise Linear Modulation (FiLM): Standard methods for incorporating subject-specific information.
  • Auto-decoding: Optimizing a subject embedding vector while keeping the main network parameters fixed.
  • Bias-terms Fine-tuning (BitFit): Fine-tuning only the bias terms of the network for a specific subject.
  • Low-Rank Adaptation (LoRA): Introducing low-rank matrices to adapt specific layers for each subject, offering parameter efficiency.

Datasets

  • CIPIC dataset: Used for single-subject HRTF interpolation experiments. This dataset contains HRTFs from 45 subjects with 1250 measurements per subject.
  • HUTUBS dataset: Used for evaluating the adaptation of a multi-subject pre-trained NF to new subjects. This dataset includes HRTFs from 94 subjects with 440 measurements per subject.

Key Insights and Ideas

  • IIR Filters Capture HRTF Modalities: The paper hypothesizes that IIR filters, with their inherent resonance and pole-zero structures, are a more natural fit for modeling the modal nature of HRTFs compared to FIR filters. This leads to more efficient representations.
  • Parameter Prediction is More Stable: Learning IIR filter parameters (like center frequencies and gains) is found to be more predictable and generalizable across directions than learning spectral magnitudes or time-domain impulse responses, especially when data is limited.
  • Parameter-Efficient Adaptation is Crucial: For personalization, methods like LoRA and BitFit significantly improve performance with fewer trainable parameters, making the adaptation process more efficient and potentially more robust with limited subject-specific data.
  • Cascaded IIR Filters Preserve Temporal Shape: Even with a phase-insensitive loss function, the learned IIR filters were shown to capture the temporal shape of the head-related impulse response, which is a desirable characteristic.

Key Results

  • Improved Upsampling Accuracy: NIIRF, especially with a larger number of IIR filters (K=32), shows improved upsampling accuracy compared to existing NF-based methods when the number of training measurements is limited. For instance, it outperforms "Mag. NF" when measurements are sparse.
  • Competitive Performance: NIIRF with K=8 filters achieves performance comparable to the "Mag. VBAP" baseline, and with K=32, it shows further improvements.
  • Effective Personalization: All explored adaptation methods (CbC, FiLM, BitFit, LoRA) significantly improve HRTF personalization compared to a generic single-subject model. NIIRF with LoRA demonstrated strong performance, particularly on unseen directions.
  • Efficiency: IIR filters require fewer coefficients than FIR filters for similar approximation quality, leading to potential reductions in computational complexity and memory footprint.

Limitations and Future Work

  • ITD Ignored: The current NIIRF model does not explicitly model the Interaural Time Difference (ITD), relying on a phase-insensitive loss. While this simplifies the problem and has been shown to work, explicitly incorporating ITD could further improve accuracy.
  • Frequency Range Limitations: The paper mentions that the frequency range of the filter parameters is fixed based on quantiles of peak/notch frequencies. This might limit flexibility for HRTFs with unusual spectral characteristics.
  • Future Work: The authors suggest exploring more advanced NF architectures and further investigating parameter-efficient fine-tuning methods for even better personalization. They also implicitly point towards incorporating ITD modeling as a natural next step.

Summary updated

Abstract

Head-related transfer functions (HRTFs) are important for immersive audio, and their spatial interpolation has been studied to upsample finite measurements. Recently, neural fields (NFs) which map from sound source direction to HRTF have gained attention. Existing NF-based methods focused on estimating the magnitude of the HRTF from a given sound source direction, and the magnitude is converted to a finite impulse response (FIR) filter. We propose the neural infinite impulse response filter field (NIIRF) method that instead estimates the coefficients of cascaded IIR filters. IIR filters mimic the modal nature of HRTFs, thus needing fewer coefficients to approximate them well compared to FIR filters. We find that our method can match the performance of existing NF-based methods on multiple datasets, even outperforming them when measurements are sparse. We also explore approaches to personalize the NF to a subject and experimentally find low-rank adaptation to be effective.
BibTeX
@inproceedings{masuyama2024niirf,
  author = {Yoshiki Masuyama and Gordon Wichern and François G. Germain and Zexu Pan and Sameer Khurana and Chiori Hori and Jonathan Le Roux},
  title = {{NIIRF: Neural IIR Filter Field for HRTF Upsampling and Personalization}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
}
Figure