PrimeK-Net: Multi-scale Spectral Learning via Group Prime-Kernel Convolutional Neural Networks for Single Channel Speech Enhancement

Zizhen Lin, Junyu Wang, R. Li, Fei Shen, Xi Xuan

Abstract

Single-channel speech enhancement is a challenging ill-posed problem focused on estimating clean speech from degraded signals. Existing studies have demonstrated the competitive performance of combining convolutional neural networks (CNNs) with Transformers in speech enhancement tasks. However, existing frameworks have not sufficiently addressed computational efficiency and have overlooked the natural multi-scale distribution of the spectrum. Additionally, the potential of CNNs in speech enhancement has yet to be fully realized. To address these issues, this study proposes a Deep Separable Dilated Dense Block (DSDDB) and a Group Prime Kernel Feedforward Channel Attention (GPFCA) module. Specifically, the DSDDB introduces higher parameter and computational efficiency to the Encoder/Decoder of existing frameworks. The GPFCA module replaces the position of the Conformer, extracting deep temporal and frequency features of the spectrum with linear complexity. The GPFCA leverages the proposed Group Prime Kernel Feedforward Network (GPFN) to integrate multi-granularity long-range, medium-range, and short-range receptive fields, while utilizing the properties of prime numbers to avoid periodic overlap effects. Experimental results demonstrate that PrimeK-Net, proposed in this study, achieves state-of-the-art (SOTA) performance on the VoiceBank+Demand dataset, reaching a PESQ score of 3.61 with only 1.41M parameters.

Introduction

Speech enhancement—the process of removing noise from a signal while preserving the speaker's voice—is a fundamental challenge in audio processing. Recently, the field has been dominated by "Two-Stage" architectures that combine Convolutional Neural Networks (CNNs) for local features and Transformers (or Conformers) for long-range dependencies. However, these methods often suffer from quadratic complexity ($O(t^2)$), making them memory-intensive for edge devices. Furthermore, standard models often ignore the multi-scale nature of speech, where features range from transient short-term patterns to persistent long-term structures.

PrimeK-Net addresses these bottlenecks by replacing heavy Transformer components with two novel, efficient modules: the Deep Separable Dilated Dense Block (DSDDB) and the Group Prime-kernel Feed-forward Channel Attention (GPFCA) module.

Overall architecture of PrimeK-Net, showing the flow from magnitude and phase inputs through the Encoder, TS-GPFCA, and specialized Decoders.

Methodology

Deep Separable Dilated Dense Block (DSDDB)

The DSDDB is an optimized version of the standard Dilated Dense Block used in encoders and decoders. By applying depth-wise separable convolutions, the authors reduced the computational complexity of these blocks to just 12.7% of the original, with almost no loss in speech quality. This allows the model to remain lightweight while maintaining deep feature extraction capabilities.

Group Prime-kernel Feed-forward Channel Attention (GPFCA)

The GPFCA module serves as the core of the network, replacing the traditional Conformer block. It utilizes a Simplified Channel Attention (SCA) mechanism to capture global information with linear complexity.

Detailed view of the GPFCA module and the Group Prime-kernel Gated Unit (GPGU), illustrating the parallel paths with prime-numbered kernel sizes (3, 11, 23, 31).

As shown in the detailed architecture, the Group Prime-kernel Feed-forward Network (GPFN) employs parallel paths with prime-numbered kernel sizes (3, 11, 23, 31). This design allows the model to perceive audio at multiple scales simultaneously. Inspired by the Multi-Period Discriminator in HiFi-GAN, using prime numbers helps the network avoid "periodic overlap effects"—interference patterns that can occur with regular, non-prime kernel spacing—effectively mimicking the long-range capability of Transformers with efficient convolutions.

Key Results and Insights

PrimeK-Net was evaluated on the VoiceBank+DEMAND dataset, a standard benchmark for speech enhancement. The model achieved a PESQ score of 3.61, setting a new state-of-the-art for its parameter class.

Key efficiency metrics include:

  • Parameters: 1.41M (compared to 2.05M for MP-SENet).
  • Compute: 44.64G FLOPs (compared to 74.52G for MP-SENet).
  • Scaling: Unlike Transformer-based models, PrimeK-Net’s memory usage scales linearly, remaining stable as audio length increases.

Conclusion

PrimeK-Net demonstrates that carefully designed CNN architectures can remain competitive with, and even outperform, heavier Transformer-based models. By leveraging the mathematical properties of prime-numbered kernels and depth-wise separable convolutions, the model achieves high-quality speech enhancement with significantly reduced computational overhead. Future work will explore applying this prime-kernel approach to multi-speaker separation and automatic speech recognition.

Summary updated

Abstract

Single-channel speech enhancement is a challenging ill-posed problem focused on estimating clean speech from degraded signals. Existing studies have demonstrated the competitive performance of combining convolutional neural networks (CNNs) with Transformers in speech enhancement tasks. However, existing frameworks have not sufficiently addressed computational efficiency and have overlooked the natural multi-scale distribution of the spectrum. Additionally, the potential of CNNs in speech enhancement has yet to be fully realized. To address these issues, this study proposes a Deep Separable Dilated Dense Block (DSDDB) and a Group Prime Kernel Feedforward Channel Attention (GPFCA) module. Specifically, the DSDDB introduces higher parameter and computational efficiency to the Encoder/Decoder of existing frameworks. The GPFCA module replaces the position of the Conformer, extracting deep temporal and frequency features of the spectrum with linear complexity. The GPFCA leverages the proposed Group Prime Kernel Feed-forward Network (GPFN) to integrate multi-granularity long-range, medium-range, and short-range receptive fields, while utilizing the properties of prime numbers to avoid periodic overlap effects. Experimental results demonstrate that PrimeK-Net, proposed in this study, achieves state-of-the-art (SOTA) performance on the VoiceBank+Demand dataset, reaching a PESQ score of 3.61 with only 1.41M parameters.
BibTeX
@misc{lin2025primek,
  author = {Zizhen Lin and Junyu Wang and R. Li and Fei Shen and Xi Xuan},
  title = {{PrimeK-Net: Multi-scale Spectral Learning via Group Prime-Kernel Convolutional Neural Networks for Single Channel Speech Enhancement}},
  year = {2025},
  eprint = {2502.19906},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.19906},
  howpublished = {arXiv},
}
Figure