Temporal Embedding in Convolutional Neural Networks for Robust Learning of Abstract Snippets

Jiajun Liu, Kun Zhao, Brano Kusy, Ji-Rong Wen, Raja JurdakarXiv 2015 · DOI · OpenReview

Introduction

Predicting daily human behavior or power consumption is inherently challenging because real-life schedules are rarely perfectly aligned. Temporal distortions—such as a commuter catching a later bus or a household shifting its peak energy usage—create misalignments in time-series data that traditional statistical models and motif-discovery algorithms struggle to resolve. This paper introduces TeNet (Temporal embedding enhanced convolutional neural Network), a novel architecture designed to automatically realign distorted time-series data to its dominant underlying patterns, enabling robust regression from partial daily sequences.

The TeNet Architecture

The core innovation of TeNet is its four-layer design, which prioritizes the discovery of abstract "snippets" or recurring behavioral routines while remaining resilient to temporal shifts.

The TeNet architecture featuring the novel temporal embedding layer followed by convolutional and pooling stages.
  1. Temporal Embedding Layer: Unlike standard convolutional layers that share weights across a sequence, this layer utilizes individualized weights for each time step. It encodes each data point alongside its immediate temporal neighbors, effectively acting as a supervised denoising step that "shifts" distorted data back into alignment.
  2. 1D Convolution and Max-Pooling: This stage functions as a suite of snippet detectors. It identifies local, recurring abstract patterns of moderate length within the realigned time-series.
  3. Sigmoid Layer: This layer serves as a non-linear feature extractor, reducing dimensionality while preserving salient information.
  4. L1-Regularized Least-Squares: The final regression layer employs L1 regularization to enforce sparsity. This ensures the model bases its predictions on the most distinct, high-level features rather than noisy signals.

Experimental Setup and Datasets

The model was evaluated across 15 individual datasets representing three distinct modalities:

  • Human Mobility (HMD/HMT): Daily traveling distance and time extracted from GPS data.
  • Household Power Consumption (HPC): High-resolution smart meter data from households in France and Australia.
  • Synthetic Data: A dataset created by artificially distorting Australian power data to specifically test the temporal embedding layer's realignment capabilities.

Performance Analysis

TeNet was benchmarked against several competitive baselines, including Support Vector Regression (with Linear, RBF, and Polynomial kernels) and Multiple Kernel Regression (MKR). The primary task was to predict total daily accumulation using only data available up to 2:00 PM.

Comparative performance of TeNet against SVR and MKR baselines across multiple accuracy and error metrics.

TeNet demonstrated clear dominance, achieving the best performance on nearly all datasets. On average, it provided a 13% increase in HitRate@20% and a 9.1% decrease in Mean Relative Error (MRE) compared to the next-best performing method. An ablation study on synthetic data further highlighted the importance of the temporal embedding layer, which reduced Mean Squared Error (MSE) from 15.5 to 6.4 when added to a standard CNN.

Key Insights and Discussion

  • Automated Realignment: Empirical evidence suggests the temporal embedding layer learns to reverse schedule shifts. For instance, if a routine is delayed, the layer learns weights that effectively push the data back to its standard temporal slot.
  • Sparse Feature Learning: Visualization of the CNN filters indicates that the network learns sparse, distinct "snippets"—abstract patterns with clear spikes and valleys—rather than dense, noisy representations.
  • Complexity Scaling: The researchers observed a strong positive correlation between TeNet's performance advantage and the entropy (complexity) of the dataset. As data becomes messier and sample sizes grow, TeNet's ability to extract abstract features allows it to scale more effectively than traditional models.

Conclusion

TeNet introduces a robust framework for time-series regression by combining individualized temporal embedding with the feature-extraction power of CNNs. By learning to "undo" temporal distortions, the model captures the underlying routines of human behavior and power consumption more accurately than traditional regression techniques. Future work may extend this architecture to general $k$-ahead forecasting and broader time-series prediction tasks.

Summary updated

Abstract

The prediction of periodical time-series remains challenging due to various types of data distortions and misalignments. Here, we propose a novel model called Temporal embedding-enhanced convolutional neural Network (TeNet) to learn repeatedly-occurring-yet-hidden structural elements in periodical time-series, called abstract snippets, for predicting future changes. Our model uses convolutional neural networks and embeds a time-series with its potential neighbors in the temporal domain for aligning it to the dominant patterns in the dataset. The model is robust to distortions and misalignments in the temporal domain and demonstrates strong prediction power for periodical time-series. We conduct extensive experiments and discover that the proposed model shows significant and consistent advantages over existing methods on a variety of data modalities ranging from human mobility to household power consumption records. Empirical results indicate that the model is robust to various factors such as number of samples, variance of data, numerical ranges of data etc. The experiments also verify that the intuition behind the model can be generalized to multiple data types and applications and promises significant improvement in prediction performances across the datasets studied.
BibTeX
@misc{liu2015temporal,
  author = {Jiajun Liu and Kun Zhao and Brano Kusy and Ji-Rong Wen and Raja Jurdak},
  title = {{Temporal Embedding in Convolutional Neural Networks for Robust Learning of Abstract Snippets}},
  year = {2015},
  eprint = {1502.05113},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1502.05113},
}
Figure