Deep Residual-Dense Lattice Network for Speech Enhancement

Mohammad Nikzad, Aaron Nicolson, Yongsheng Gao, Jun Zhou, Kuldip K. Paliwal, Fanhua Shang

Introduction

Speech enhancement is a critical task in audio processing, aiming to isolate clean speech from noisy backgrounds. While deep learning has significantly advanced the field, current architectures often face a trade-off between feature preservation and computational efficiency. Residual Networks (ResNets) facilitate training deep models through skip connections but can suffer from "feature diminution," where shallow layer information is lost over depth. Conversely, Densely Connected Networks (DenseNets) preserve features through concatenation but lead to a "parameter explosion" as input sizes grow linearly with depth.

The Residual-Dense Lattice Network (RDL-Net) introduces a novel architecture that balances these two extremes. By organizing convolutional units in a triangular lattice, RDL-Net achieves state-of-the-art performance with significantly lower memory and computational requirements than traditional deep learning models.

The Residual-Dense Lattice (RDL) Architecture

The fundamental building block of this network is the RDL block. Unlike standard sequential layers, the RDL block arranges units in a two-dimensional triangular lattice defined by height ($h$) and length ($l$).

The triangular lattice structure of an RDL block, showing the arrangement of convolutional units and hybrid connectivity.

This topology utilizes three distinct types of connectivity to optimize information flow:

  1. Local Dense Links: Within a block, features are concatenated along the "height" of the lattice. By capping the height, the authors prevent the linear growth of input sizes seen in standard DenseNets.
  2. Local Residual Links: Summation occurs across the "length" of the lattice, stabilizing gradient flow and allowing for deeper architectures.
  3. Global Dense Links: High-level features are shared across the entire network by concatenating outputs between different RDL blocks.

To further optimize efficiency, the network employs 1D causal dilated convolutions and varies kernel sizes and output dimensions based on the unit's position within the lattice.

Efficiency and Performance

The primary advantage of RDL-Net is its remarkable parameter efficiency. The triangular lattice structure ensures that feature re-usage is targeted rather than exhaustive, capturing essential information without redundant connectivity.

Comparison of validation error versus parameter count, demonstrating that RDL-Net achieves superior performance with approximately 2x fewer parameters than ResNet.

Empirical results show that an RDL-Net with 1 million parameters matches the validation error of a ResNet with 2 million parameters. Furthermore, the model requires approximately four times fewer Floating Point Operations (FLOPs) than traditional ResNets to achieve comparable performance, making it highly suitable for resource-constrained environments like hearing aids or mobile devices.

Experimental Evaluation

RDL-Net was rigorously tested using large-scale datasets, including LibriSpeech, VCTK, and TIMIT for training, and the Voice Bank + DEMAND corpus for benchmarking. The model consistently outperformed established baselines across all standard objective metrics, including PESQ (perceptual quality) and STOI (intelligibility).

Qualitative spectrogram analysis further confirmed that RDL-Net effectively suppresses background noise while minimizing formant distortion, thereby preserving the natural characteristics of the human voice. Even the smallest RDL-Net configurations (e.g., 0.53M parameters) remained competitive with much larger traditional models.

Conclusion

The Residual-Dense Lattice Network represents a significant step forward in efficient deep learning for audio. By rethinking network topology through the lens of a triangular lattice, the authors demonstrate that strategic connectivity is more effective than sheer model size. Future research may extend this lattice-based approach to other domains such as computer vision and real-time edge computing applications.

Summary updated

BibTeX
@inproceedings{nikzad2020deep,
  author = {Mohammad Nikzad and Aaron Nicolson and Yongsheng Gao and Jun Zhou and Kuldip K. Paliwal and Fanhua Shang},
  title = {{Deep Residual-Dense Lattice Network for Speech Enhancement}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i05.6377},
}
Figure