Multivariate Times Series Classification Using Multichannel CNN

YongKyung Oh

Introduction

Multivariate Time Series Classification (MTSC) is a critical task in modern data science, appearing in domains ranging from healthcare (EEG/ECG monitoring) to industrial IoT (sensor arrays). Despite the success of deep learning in other fields, MTSC faces two significant challenges: architecture mismatch and data scarcity. Most high-performance neural networks, particularly those in computer vision, are optimized for 2D spatial data rather than 1D temporal sequences. Furthermore, training complex models from scratch requires massive labeled datasets, which are often unavailable in specialized fields.

This research proposes a novel framework to bridge this gap by "re-skinning" time series data into a format compatible with state-of-the-art computer vision models. By transforming temporal data into multichannel "analogous images," the authors leverage transfer learning from the ImageNet domain to boost classification accuracy, particularly on small-scale datasets.

Methodology

The proposed method utilizes a two-step pipeline designed to repurpose the power of pretrained convolutional neural networks (CNNs).

1. Image Transformation

Instead of processing raw numerical sequences, the method transforms each feature of a multivariate time series into a grayscale "analogous image." For a dataset with $M$ features, $M$ corresponding grayscale images are generated. These images are then stacked as channels, creating a single multichannel input. For example, a dataset with six sensors would produce a 6-channel image, analogous to how a standard photograph consists of Red, Green, and Blue (RGB) channels.

Transformation of multivariate time series data into multichannel analogous images.

2. Multichannel Transfer Learning

The researchers adapted a standard ResNet50 architecture for this task. While the original ResNet50 is designed for 3-channel RGB inputs, this version is modified to accept $M$ channels. The model is initialized with weights pretrained on ImageNet and then fine-tuned on the generated time series images. This approach allows the model to utilize universal low-level features (edges, textures, and curves) learned from natural images to identify patterns in signal data.

Experimental Setup

The framework was evaluated using five datasets from the UEA (University of East Anglia) multivariate time series classification archive, specifically chosen for their limited training sizes:

  • Epilepsy (EP): 3 features, 4 classes.
  • EthanolConcentration (EC): 3 features, 4 classes.
  • RacketSports (RS): 6 features, 4 classes.
  • StandWalkJump (SWJ): 4 features, 3 classes.
  • UWaveGestureLibrary (UWG): 3 features, 8 classes.

Key Results and Insights

The performance of the proposed method was compared against several established baselines, including MLP, FCN, 1D-ResNet, Time-CNN, and InceptionTime.

Performance comparison (Weighted F1 and Rank) of the proposed method against baseline models across five UEA datasets.

The Power of Transfer Learning

A critical finding was the impact of pretraining. The "Proposed (False)" model, which used random initialization, performed poorly with an average rank of 6.0. In contrast, the "Proposed (True)" model, utilizing ImageNet weights, achieved the highest average rank (2.4) across all datasets. This confirms that features learned from visual data are highly transferable to the temporal signal domain.

Performance Highlights

  • Top Ranking: The proposed method secured the #1 spot in weighted F1-score for both the Epilepsy (0.987) and EthanolConcentration (0.271) datasets.
  • Consistency: While models like FCN performed better on specific datasets (e.g., RacketSports), the proposed transfer learning approach was the most consistent performer overall.
  • Cross-Sensor Correlation: By mapping features to channels, the CNN filters can simultaneously learn temporal patterns within a single sensor and correlations between different sensors.

Limitations and Future Work

While the results demonstrate the efficacy of transfer learning for MTSC, the authors identify several areas for improvement:

  1. Computational Overhead: Transforming time series into high-resolution images and processing them through a deep ResNet50 is computationally intensive, potentially limiting deployment on edge devices like wearables.
  2. Latency: The time required for image generation and the forward pass must be optimized for real-time applications.
  3. Scalability: Future research will expand testing to a broader range of MTSC benchmarks to further validate the universality of this visual-to-temporal transfer learning approach.

Summary updated

Abstract

Multivariate time series classification is an important and demanding task in sequence data mining. We focus on the multichannel representation of the time series and its corresponding convolutional neural network (CNN) classifier. The proposed method transforms multivariate time series into multichannel analogous image and it is fed into a pretrained multichannel CNN with transfer learning. To verify the efficacy of the proposed method, we compared it with recent deep learning-based time series classification models on five datasets with small amounts of training data. The results indicate that the proposed method provides improved performance on average compared with the other methods when incorporated with transfer learning.
BibTeX
@inproceedings{oh2022multivariate,
  author = {YongKyung Oh},
  title = {{Multivariate Times Series Classification Using Multichannel CNN}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.24963/ijcai.2022/835},
}
Figure