Deep Residual-Dense Lattice Network for Speech Enhancement

Mohammad Nikzad, Aaron Nicolson, Yongsheng Gao, Jun Zhou, Kuldip K. Paliwal, Fanhua Shang

Introduction

Deep learning for speech enhancement has traditionally been dominated by two architectural paradigms: ResNets and DenseNets. ResNets utilize residual links to facilitate gradient flow in deep networks but can suffer from "feature diminution," where information from early layers is progressively lost. Conversely, DenseNets ensure total feature re-usage by concatenating all previous layer outputs, but this leads to a "parameter explosion" as input sizes grow linearly with depth, making them computationally expensive.

The Residual-Dense Lattice Network (RDL-Net) addresses this trade-off by introducing a novel triangular lattice topology. This architecture provides the feature-rich exploration of DenseNets while maintaining the parameter efficiency required for real-time speech processing.

The Residual-Dense Lattice (RDL) Architecture

The core innovation of RDL-Net is the Residual-Dense Lattice (RDL) block. Instead of a simple chain of layers, convolutional units are arranged into a triangular lattice structure.

The Residual-Dense Lattice (RDL) block topology, showing the triangular arrangement of convolutional units and the flow of information through local dense and residual links.

This topology is characterized by several key design choices:

  • Lattice Geometry: The block is split into a "left triangle" for feature formation and a "right triangle" for feature amalgamation. This geometric constraint naturally limits the number of inputs to any single unit, preventing the parameter bloat seen in standard DenseNets.
  • Local Dense Aggregations: Within the lattice, units concatenate a subset of previous outputs, enabling effective feature re-usage without overwhelming input dimensions.
  • Local Residual Links: Summation-based residual links are integrated within the lattice to stabilize training and improve gradient flow.
  • Global Dense Links: The network employs links between entire RDL blocks, allowing the model to re-use features across the entire depth of the network.
  • Causal Dilated Convolutions: Each unit utilizes 1D causal dilated convolutions, which are ideal for audio processing as they capture long-term temporal dependencies without looking into the future.

RDL-Net is integrated into the Deep Xi framework, where it estimates the a priori Signal-to-Noise Ratio (SNR) to filter noisy speech.

Experimental Results and Efficiency

The RDL-Net was evaluated against standard benchmarks, including the Voice Bank + DEMAND dataset, using a mix of clean speech from LibriSpeech and TIMIT, and noise from RSG-10 and Urban Sound.

Performance comparison of RDL-Net against baseline architectures. (a-b) Training and validation error over epochs. (c-d) Trade-off between validation error and computational resources (parameters and FLOPs).

RDL-Net consistently outperformed ResNets, DenseNets, and specialized architectures like SEGAN and Metric-GAN. On the Voice Bank + DEMAND set, RDL-Net achieved a PESQ score of 3.02, significantly higher than Metric-GAN's 2.86.

Beyond raw performance, the RDL-Net demonstrates exceptional efficiency. As illustrated in the figure above, the model requires approximately 2x fewer parameters and 4x fewer FLOPs (floating-point operations) to reach the same validation error as a standard ResNet. This "parameter economy" proves that intelligent connectivity can be more effective than simply increasing model size.

Conclusion

The RDL-Net represents a significant step forward in efficient deep learning for speech enhancement. By leveraging a triangular lattice topology, it achieves state-of-the-art results with a fraction of the computational cost of traditional architectures. While currently focused on magnitude-spectrum enhancement, the authors suggest that this lattice structure is a general-purpose architecture suitable for other domains, including computer vision and image denoising.

Summary updated

BibTeX
@inproceedings{nikzad2020deep,
  author = {Mohammad Nikzad and Aaron Nicolson and Yongsheng Gao and Jun Zhou and Kuldip K. Paliwal and Fanhua Shang},
  title = {{Deep Residual-Dense Lattice Network for Speech Enhancement}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i05.6377},
}
Figure