Domestic Activity Clustering from Audio via Depthwise Separable Convolutional Autoencoder Network

Yanxiong Li, Wenchang Cao, Konstantinos Drossos, Tuomas VirtanenarXiv 2022 · arXiv

TL;DR

This paper introduces the Depthwise Separable Convolutional Autoencoder Network (DSCAN), a lightweight deep learning framework designed to cluster unlabeled domestic audio clips into distinct activity categories. By combining depthwise separable convolutions with a joint clustering-reconstruction loss, the model achieves state-of-the-art clustering accuracy while dramatically reducing computational complexity and memory requirements compared to previous heavy deep models.

What Problem Does This Paper Address?

Smart home and safety monitoring systems (such as those designed for elderly care) typically rely on video cameras or wearable sensors to track human activity. However, video monitoring raises significant privacy concerns and suffers from blind spots, while wearables require consistent user compliance.

Audio-based monitoring offers a non-intrusive, privacy-preserving alternative. While Acoustic Scene Classification (ASC) has advanced rapidly, it heavily relies on large, manually labeled datasets—which are expensive and time-consuming to produce. To bypass this bottleneck, the authors tackle domestic activity clustering, an unsupervised task where the system must group similar audio clips (e.g., cooking, dishwashing, or vacuuming) without any prior knowledge of their ground-truth labels.

Prior deep learning approaches to this problem rely on heavyweight architectures like capsule networks, making them impractical for resource-constrained edge devices. This paper aims to bridge the gap between high clustering performance and low computational overhead.

Methods

The authors propose the Depthwise Separable Convolutional Autoencoder Network (DSCAN), which consists of three main components: an encoder, a decoder, and a clustering layer.

Fig. 1. The framework of the proposed DSCAN.
  1. Lightweight Encoder: Instead of standard convolutions, the encoder relies on Depthwise Separable Convolutions (DSC). DSC splits standard convolution into a depthwise step (filtering per channel) and a pointwise step (combining channels), theoretically reducing the computational load by 8 to 9 times for $3\times3$ kernels. The encoder also utilizes residual skip connections to preserve feature maps and replaces complex capsule modules with a simple, efficient fully-connected layer.
  2. Decoder: Mirroring the encoder, the decoder uses transposed convolutional blocks with residual structures to reconstruct the original log-mel spectrogram from the learned low-dimensional embedding.
  3. Clustering Layer & Joint Loss: The network is optimized using a joint loss function: $$L_J = L_r + \beta L_c$$ where $L_r$ is the mean squared error reconstruction loss, and $L_c$ is a clustering loss based on the Kullback-Leibler (KL) divergence between the model's soft cluster assignments (measured via a Student’s t-distribution) and a high-confidence target distribution. The hyperparameter $\beta$ balances the two tasks to prevent the embedding space from distorting.

Datasets

The model was evaluated using two datasets:

  • Development Dataset: Compiled from Task 1A of the DCASE 2019 challenge (12,960 clips across 9 acoustic scenes) to tune the loss-balancing hyperparameter $\beta$.
  • Evaluation Dataset: Task 5 of the DCASE-2018 challenge, which is a derivative of the SINS dataset. It contains 72,984 10-second audio clips recorded in a home environment across 9 daily activities: Dishwashing, Cooking, Absence, Eating, Other, Vacuum cleaning, Social activity, Watching TV, and Working.

Key Insights and Ideas

  • The Power of Joint Optimization: Training the autoencoder purely for reconstruction doesn't guarantee well-separated clusters, while training purely for clustering can warp the embedding space. Jointly optimizing both tasks ensures the embeddings retain meaningful time-frequency properties while aligning into distinct clusters.
  • Simplification Over Complexity: While recent state-of-the-art methods like CCAN rely on complex, computationally heavy capsule modules to capture spatial/structural relationships, this paper demonstrates that a carefully tuned fully-connected layer combined with DSC can achieve superior results with a fraction of the footprint.
  • Hyperparameter Sensitivity: The authors found that setting $\beta = 0.3$ yielded the optimal balance. Deviating too far from this value caused a steady decline in both clustering accuracy and normalized mutual information.

Key Results

The DSCAN-based method was compared against five deep-model baselines, including Stacked Autoencoders (SA), Convolutional Autoencoders (CoAN), and the recent Convolutional Capsule Autoencoder Network (CCAN).

  • Clustering Performance: DSCAN achieved a Clustering Accuracy (CA) of 63.64% and a Normalized Mutual Information (NMI) score of 54.46%, outperforming the state-of-the-art CCAN by 1.73% and 0.62% absolute gains, respectively.
  • Efficiency Gains:
  • Model Size (MS): DSCAN requires only 72.4 K parameters, compared to CCAN's 725.3 K and SA's massive 1842.9 K.
  • Computational Complexity: DSCAN required 1048.8 K MACs (Multiply-Accumulate operations), making it significantly lighter than all other tested deep models.

A t-SNE visualization of the resulting embedding space shows that while most activities form distinct clusters, some overlap still occurs, particularly with the ambiguous "Other" category.

Fig. 5. Visualization of clustering results obtained by our method.

Limitations and Future Work

Despite its strong performance, the authors highlight several remaining challenges in domestic audio clustering:

  1. Data Imbalance: Real-world domestic data is highly unbalanced. For instance, the "Absence" class contains over 18,000 clips, while "Vacuum cleaning" has fewer than 1,000, which naturally skews unsupervised clustering behavior.
  2. Acoustic Overlap: Different activities often share similar time-frequency properties (e.g., background noise or ambient sounds), leading to overlapping feature distributions that confuse the model.

In future work, the authors plan to deploy this model directly onto low-computing edge terminals and expand its application to outdoor acoustic scenes like roads and transit stations.

Summary updated

Abstract

Automatic estimation of domestic activities from audio can be used to solve many problems, such as reducing the labor cost for nursing the elderly people. This study focuses on solving the problem of domestic activity clustering from audio. The target of domestic activity clustering is to cluster audio clips which belong to the same category of domestic activity into one cluster in an unsupervised way. In this paper, we propose a method of domestic activity clustering using a depthwise separable convolutional autoencoder network. In the proposed method, initial embeddings are learned by the depthwise separable convolutional autoencoder, and a clustering-oriented loss is designed to jointly optimize embedding refinement and cluster assignment. Different methods are evaluated on a public dataset (a derivative of the SINS dataset) used in the challenge on Detection and Classification of Acoustic Scenes and Events (DCASE) in 2018. Our method obtains the normalized mutual information (NMI) score of 54.46%, and the clustering accuracy (CA) score of 63.64%, and outperforms state-of-the-art methods in terms of NMI and CA. In addition, both computational complexity and memory requirement of our method is lower than that of previous deep-model-based methods. Codes: https://github.com/vinceasvp/domestic-activity-clustering-from-audio
BibTeX
@misc{li2022domestic,
  author = {Yanxiong Li and Wenchang Cao and Konstantinos Drossos and Tuomas Virtanen},
  title = {{Domestic Activity Clustering from Audio via Depthwise Separable Convolutional Autoencoder Network}},
  year = {2022},
  eprint = {2208.02406},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2208.02406},
}
Figure