J-Net: Randomly weighted U-Net for audio source separation

Bo-Wen Chen, Yen-Min Hsu, Hung-Yi LeearXiv 2019 · OpenReview

Abstract

Several results in the computer vision literature have shown the potential of randomly weighted neural networks. While they perform fairly well as feature extractors for discriminative tasks, a positive correlation exists between their performance and their fully trained counterparts. According to these discoveries, we pose two questions: what is the value of randomly weighted networks in difficult generative audio tasks such as audio source separation and does such positive correlation still exist when it comes to large random networks and their trained counterparts? In this paper, we demonstrate that the positive correlation still exists. Based on this discovery, we can try out different architecture designs or tricks without training the whole model. Meanwhile, we find a surprising result that in comparison to the non-trained encoder (down-sample path) in Wave-U-Net, fixing the decoder (up-sample path) to random weights results in better performance, almost comparable to the fully trained model.

Introduction

Training deep neural networks for generative tasks like audio source separation—splitting a mixture into components like vocals and accompaniment—is computationally intensive. In computer vision, it has been observed that "randomly weighted" networks can act as effective feature extractors and, crucially, their performance correlates with that of their fully trained versions. This paper investigates whether this correlation holds for complex audio tasks, potentially allowing researchers to use untrained models as a proxy to accelerate architecture search.

Methodology

The researchers utilized Wave-U-Net, a 1D adaptation of the U-Net architecture for time-domain waveforms, as the baseline. To test their hypotheses, they introduced several variants:

  • J-Net: A model where the encoder (down-sampling path) is fixed with random weights, and only the decoder is trained.
  • MultiRes Block: Inception-like blocks with multiple filter sizes to capture multi-resolution audio features.
  • Res path: Replacing direct skip connections with non-linear CNN layers to bridge the semantic gap between features.
  • Progressive Learning & Identity Loss: Techniques to refine separation and preserve domain content.
The proposed architecture incorporates MultiRes blocks and a parameter-sharing mirror for source separation.

The models were evaluated on the MUSDB18 and CCMixter datasets, processed at 22,050Hz in stereo.

Results and Discussion

The study confirms a strong positive correlation between the performance of J-Net (untrained encoder) and the fully trained Wave-U-Net, particularly in vocal separation. This validates the use of random weights as a proxy for evaluating architectural innovations like MultiRes blocks without full training.

Scatter plot showing the positive correlation between the SDR of randomly-weighted models and their trained counterparts for vocal separation.

Performance Highlights

  • State-of-the-Art Results: The MultiRes U-Net ($U3_{10}$) achieved a median Signal-to-Distortion Ratio (SDR) of 4.84 for vocals, surpassing the baseline U1 (4.48) and the previous state-of-the-art M4 (4.46).
  • The Power of Skip Connections: A surprising finding was the effectiveness of the "L-Net" (random decoder). A model with a random decoder ($L_{last-3}$) achieved a median vocal SDR of 3.93. This suggests that skip connections in U-Net architectures allow the decoder to function primarily as a combiner of high-resolution encoder features rather than a complex reconstructor.

Conclusion

This research demonstrates that architectural evaluation in audio source separation can be significantly accelerated by using randomly weighted encoders. While a "ceiling effect" was noted in accompaniment separation due to its complexity, the vocal separation results provide a clear path for faster model iteration. Future work will explore the mechanics of skip connections and the applicability of this framework to other generative audio tasks.

Summary updated

Abstract

Several results in the computer vision literature have shown the potential of randomly weighted neural networks. While they perform fairly well as feature extractors for discriminative tasks, a positive correlation exists between their performance and their fully trained counterparts. According to these discoveries, we pose two questions: what is the value of randomly weighted networks in difficult generative audio tasks such as audio source separation and does such positive correlation still exist when it comes to large random networks and their trained counterparts? In this paper, we demonstrate that the positive correlation still exists. Based on this discovery, we can try out different architecture designs or tricks without training the whole model. Meanwhile, we find a surprising result that in comparison to the non-trained encoder (down-sample path) in Wave-U-Net, fixing the decoder (up-sample path) to random weights results in better performance, almost comparable to the fully trained model.
BibTeX
@misc{chen2019j,
  author = {Bo-Wen Chen and Yen-Min Hsu and Hung-Yi Lee},
  title = {{J-Net: Randomly weighted U-Net for audio source separation}},
  year = {2019},
  eprint = {1911.12926},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1911.12926},
}
Figure