Domestic activities clustering from audio recordings using convolutional capsule autoencoder network

Ziheng Lin, Yanxiong Li, Zhangjin Huang, Wenhao Zhang, Yufeng Tan, Yi‐Chun Chen, Qianhua He

Abstract

Recent efforts have been made on domestic activities classification from audio recordings, especially the works submitted to the challenge of DCASE (Detection and Classification of Acoustic Scenes and Events) since 2018. In contrast, few studies were done on domestic activities clustering, which is a newly emerging problem. Domestic activities clustering from audio recordings aims at merging audio clips which belong to the same class of domestic activity into a single cluster. Domestic activities clustering is an effective way for unsupervised estimation of daily activities performed in home environment. In this study, we propose a method for domestic activities clustering using a convolutional capsule autoencoder network (CCAN). In the method, the deep embeddings are learned by the autoencoder in the CCAN, while the deep embeddings which belong to the same class of domestic activities are merged into a single cluster by a clustering layer in the CCAN. Evaluated on a public dataset adopted in DCASE-2018 Task 5, the results show that the proposed method outperforms state-of-the-art methods in terms of the metrics of clustering accuracy and normalized mutual information.

Introduction

Most research in acoustic scene analysis focuses on classification, which requires extensive pre-labeled data. However, labeling massive amounts of data is expensive and prone to error. Domestic activity clustering addresses this by automatically grouping raw audio from home environments without manual labels. While Convolutional Neural Networks (CNNs) are standard, they suffer from information loss due to pooling layers and struggle to represent complex spatial relationships between sound features using only scalar outputs.

Proposed Method

The proposed Convolutional Capsule Autoencoder Network (CCAN) is a hybrid architecture designed to learn "cluster-friendly" deep embeddings.

The CCAN architecture featuring an encoder with convolutional layers and capsules, a decoder for reconstruction, and a clustering layer.

The CCAN consists of three primary components:

  • Encoder: Extracts basic patterns from Mel-frequency cepstral coefficients (MFCCs) using standard convolutional layers, followed by Primary and Class Capsules. These capsules use vectors to preserve the spatial hierarchy and relationships between audio components.
  • Decoder: Reconstructs the original MFCCs from capsule outputs to ensure learned embeddings accurately represent the input data.
  • Clustering Layer: Sits atop the class capsules and uses a Student’s t-distribution to calculate cluster membership probabilities.

The model is trained using a joint loss function that combines reconstruction loss with Kullback-Leibler divergence to ensure clusters are distinct and tight.

Experimental Evaluation

The model was evaluated using the DCASE-2018 Task 5 dataset (derived from the SINS dataset), comprising 72,984 10-second audio clips across 9 categories, including Absence, Cooking, Dishwashing, Eating, Social Activity, Vacuum Cleaning, Watching TV, Working, and Others.

Results and Discussion

The CCAN demonstrates that vector-based Capsule Networks better recognize temporal and frequency relationships compared to scalar-based CNNs. By hybridizing CNNs for low-level feature extraction with capsules for hierarchical organization, the model achieves superior unsupervised performance.

Comparison of Clustering Accuracy (CA) and Normalized Mutual Information (NMI) between the proposed CCAN and baseline methods.

As shown in the results, the proposed CCAN achieved a Clustering Accuracy (CA) of 61.91%, representing an absolute gain of 7.10% to 18.16% over baseline methods like Stacked Autoencoders (SA) and LSTM-based models. Furthermore, the CCAN reached a Normalized Mutual Information (NMI) of 53.84%, outperforming the next best method by over 7%. Visualization using t-SNE plots confirmed that while most activities clustered well, some categories like "Others" were frequently confused with "Absence," highlighting the difficulty of defining background noise.

Conclusion

The CCAN provides a robust framework for unsupervised domestic activity estimation. While challenges remain in distinguishing very similar-sounding activities and handling data imbalance, the proposed method significantly advances the state-of-the-art. Future work will explore the impact of specific capsule components and broader activity sets.

Summary updated

Abstract

Recent efforts have been made on domestic activities classification from audio recordings, especially the works submitted to the challenge of DCASE (Detection and Classification of Acoustic Scenes and Events) since 2018. In contrast, few studies were done on domestic activities clustering, which is a newly emerging problem. Domestic activities clustering from audio recordings aims at merging audio clips which belong to the same class of domestic activity into a single cluster. Domestic activities clustering is an effective way for unsupervised estimation of daily activities performed in home environment. In this study, we propose a method for domestic activities clustering using a convolutional capsule autoencoder network (CCAN). In the method, the deep embeddings are learned by the autoencoder in the CCAN, while the deep embeddings which belong to the same class of domestic activities are merged into a single cluster by a clustering layer in the CCAN. Evaluated on a public dataset adopted in DCASE-2018 Task 5, the results show that the proposed method outperforms state-of-the-art methods in terms of the metrics of clustering accuracy and normalized mutual information.
BibTeX
@misc{lin2021domestic,
  author = {Ziheng Lin and Yanxiong Li and Zhangjin Huang and Wenhao Zhang and Yufeng Tan and Yi‐Chun Chen and Qianhua He},
  title = {{Domestic activities clustering from audio recordings using convolutional capsule autoencoder network}},
  year = {2021},
  eprint = {2105.03583},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2105.03583},
}
Figure