Rubik's Cube Operator: A Plug And Play Permutation Module for Better Arranging High Dimensional Industrial Data in Deep Convolutional Processes

Luoxiao Yang, Zhong Zheng, Zijun ZhangarXiv 2022 · OpenReview

Abstract

The convolutional neural network (CNN) has been widely applied to process industrial data tensors, which integrate records from spatial, temporal, and system dynamics aspects. However, unlike images, information in these tensors is not necessarily spatially ordered, rendering direct CNN application ineffective. We propose the Rubik's Cube Operator (RCO), a plug-and-play module that adaptively permutes data organization to an optimal order before CNN processing. RCO maintains $K$ binary and right stochastic permutation matrices, updated via gradient-based optimization using Gumbel-Softmax reparameterization. A soft regularization loss ensures feature diversity. Experiments on wind power and speed prediction demonstrate that RCO significantly improves CNN performance across multiple datasets.

The Challenge of Unordered Industrial Data

Convolutional Neural Networks (CNNs) are designed for data with inherent spatial correlations, such as images where neighboring pixels are physically related. In industrial contexts—such as monitoring wind farms—data is typically organized into high-dimensional tensors representing time, sensor types (system dynamics), and physical locations.

Visualization of an industrial data tensor showing the relationship between system dynamics and spatial/temporal axes.

The primary difficulty is that the arrangement of these attributes is often arbitrary. For instance, placing "Wind Speed" adjacent to "Generator Temperature" in a tensor might not be the most effective way for a CNN kernel to extract meaningful correlations. Standard CNN kernels operate on local neighborhoods; if the data arrangement is suboptimal, the network struggles to identify features. Currently, there is no standard automated method to learn the optimal "spatial" arrangement for such non-image data.

The Rubik's Cube Operator (RCO)

To bridge this gap, we introduce the Rubik's Cube Operator (RCO). This module acts as a preprocessing layer that learns to reshuffle the input tensor into a format more conducive to convolutional feature extraction.

The RCO adaptively permutes attributes across different axes to find an optimal data arrangement for downstream CNNs.

Technical Innovations

  1. Differentiable Permutation: Since discrete permutations are non-differentiable, RCO utilizes the Gumbel-Softmax trick. This allows the permutation order to be learned through standard backpropagation alongside the rest of the network.
  2. Right Stochastic Matrices: Unlike traditional doubly stochastic matrices that ensure each feature appears exactly once, RCO uses right stochastic matrices. This allows important features to be repeated in the reshuffled tensor, enabling the CNN to better exploit shift invariance.
  3. Soft Regularization Loss: To prevent the model from collapsing onto a few dominant features and overfitting, a specialized loss function encourages diversity in the permuted output.

Experimental Evaluation

The RCO was evaluated on four real-world datasets from the renewable energy sector, involving up to 33 wind turbines and various parameters like wind power, speed, direction, and vibration data. The primary tasks were Wind Power Prediction (WPP) and Wind Speed Prediction (WSP).

Key Results

  • Accuracy Improvements: RCO-enhanced models consistently outperformed standard CNN-LSTM baselines. In WPP tasks on Dataset 2, the average Root Mean Square Error (RMSE) was reduced from 276.21 to 258.71.
  • Temporal Consistency: The performance gains were stable across different prediction horizons, including 1-step, 3-step, and 6-step ahead forecasts.
  • Operational Efficiency: While training time increased slightly (from ~7.3s to ~9.4s per epoch), the inference time remained nearly identical to the baseline, making it suitable for real-time industrial deployment.

Conclusion and Future Directions

The RCO demonstrates that for industrial data, the "best" order is the one that simplifies the task for the downstream CNN. By allowing for feature redundancy and learning permutations end-to-end, RCO provides a robust, plug-and-play solution for high-dimensional data processing.

Future work will focus on the sensitivity of the soft regularization hyperparameter and scaling the operator to systems with thousands of sensors, such as large-scale chemical plants or complex IoT arrays in manufacturing.

Summary updated

Abstract

The convolutional neural network (CNN) has been widely applied to process the industrial data based tensor input, which integrates data records of distributed industrial systems from the spatial, temporal, and system dynamics aspects. However, unlike images, information in the industrial data based tensor is not necessarily spatially ordered. Thus, directly applying CNN is ineffective. To tackle such issue, we propose a plug and play module, the Rubik's Cube Operator (RCO), to adaptively permutate the data organization of the industrial data based tensor to an optimal or suboptimal order of attributes before being processed by CNNs, which can be updated with subsequent CNNs together via the gradient-based optimizer. The proposed RCO maintains K binary and right stochastic permutation matrices to permutate attributes of K axes of the input industrial data based tensor. A novel learning process is proposed to enable learning permutation matrices from data, where the Gumbel-Softmax is employed to reparameterize elements of permutation matrices, and the soft regularization loss is proposed and added to the task-specific loss to ensure the feature diversity of the permuted data. We verify the effectiveness of the proposed RCO via considering two representative learning tasks processing industrial data via CNNs, the wind power prediction (WPP) and the wind speed prediction (WSP) from the renewable energy domain. Computational experiments are conducted based on four datasets collected from different wind farms and the results demonstrate that the proposed RCO can improve the performance of CNN based networks significantly.
BibTeX
@misc{yang2022rubik,
  author = {Luoxiao Yang and Zhong Zheng and Zijun Zhang},
  title = {{Rubik's Cube Operator: A Plug And Play Permutation Module for Better Arranging High Dimensional Industrial Data in Deep Convolutional Processes}},
  year = {2022},
  eprint = {2203.12921},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2203.12921},
}
Figure