Abstract
A single dataset could hide a significant number of relationships among its feature set. Learning these relationships simultaneously avoids the time complexity associated with running the learning algorithm for every possible relationship, and affords the learner with an ability to recover missing data and substitute erroneous ones by using available data. In our previous research, we introduced the gate-layer autoencoders (GLAEs), which offer an architecture that enables a single model to approximate multiple relationships simultaneously. GLAE controls what an autoencoder learns in a time series by switching on and off certain input gates, thus, allowing and disallowing the data to flow through the network to increase network's robustness. However, GLAE is limited to binary gates. In this article, we generalize the architecture to weighted gate layer autoencoders (WGLAE) through the addition of a weight layer to update the error according to which variables are more critical and to encourage the network to learn these variables. This new weight layer can also be used as an output gate and uses additional control parameters to afford the network with abilities to represent different models that can learn through gating the inputs. We compare the architecture against similar architectures in the literature and demonstrate that the proposed architecture produces more robust autoencoders with the ability to reconstruct both incomplete synthetic and real data with high accuracy.
1. Introduction
In real-world applications involving high-dimensional time-series data, such as Electroencephalography (EEG), data integrity is frequently compromised by sensor failure or environmental noise. Traditional methods for data imputation often require building separate estimators for every possible combination of missing variables, a process that scales poorly as the number of features increases.
While standard Autoencoders (AEs) are effective at data compression, they do not inherently prioritize the relationships between specific variables necessary for robust reconstruction. This paper introduces the Weighted Gate Layer Autoencoder (WGLAE), an evolution of the Gate-Layer Autoencoder (GLAE). By incorporating a dynamic weighting mechanism, WGLAE allows a single model to learn universal inter-dependencies across all variables simultaneously, providing a scalable and robust solution for recovering missing signals.
2. Methods and Architecture
The WGLAE architecture enhances the classic autoencoder by introducing three primary control modules that manage data flow and error propagation.
2.1 The Gating Mechanism
At the core of the architecture is the Input Gate, managed by a Gate Controller. During training, the controller applies a binary mask to "switch off" specific input variables. This forces the network to treat these masked variables as dependent variables (DVs) that must be reconstructed using the remaining "on" independent variables (IDVs).
The selection of these variables is governed by Algorithm 1, which allows for either sequential or random masking strategies, ensuring the network encounters a diverse set of "missing data" scenarios.
2.2 Error Weighting and Control
The defining innovation of WGLAE is the Error Weight Generator. Unlike previous models that treat all reconstruction errors equally, WGLAE assigns a weight ($\omega$) to the error of each variable.
By weighting the error of "unknown" (masked) variables more heavily than "known" variables, the backpropagation process is forced to prioritize the dependencies required to recover missing data. Additionally, a Network Controller organizes training into sessions, lessons, and curricula, systematically traversing all variables to ensure comprehensive relationship learning.
3. Experimental Evaluation
The authors evaluated WGLAE across three distinct types of data:
- Dataset I (Synthetic): A 7-channel simulated EEG dataset designed to mimic volume conduction.
- Dataset II (Real-world EEG): 46-channel EEG data from 59 subjects performing a Stroop test.
- Dataset III (SSVEP): Two datasets containing 8-channel Steady-State Visually Evoked Potential signals commonly used in Brain-Computer Interfaces.
4. Key Results
WGLAE demonstrated significant improvements in both reconstruction accuracy and stability:
- Robustness to Data Loss: In tests on 46-channel real EEG data, standard AEs saw error rates increase by 4.5x when moving from 1 to 9 missing channels. In contrast, WGLAE’s error remained nearly flat.
- Downstream Task Performance: In an EEG-based biometric identification task, WGLAE achieved 100% accuracy on clean data and maintained 99.84% accuracy even with two missing channels.
- Transfer Learning: Features learned by WGLAE proved highly portable; when applied to new datasets, the model required approximately 60% fewer training epochs to reach peak accuracy compared to standard AEs.
5. Conclusion
The Weighted Gate Layer Autoencoder provides a robust framework for learning complex inter-dependencies in multi-channel data. By prioritizing the reconstruction of masked variables through dynamic weighting, it overcomes the limitations of binary gating and standard compression-focused autoencoders. Future work will explore intelligent variable selection based on sensor proximity and signal characteristics to further optimize the learning curriculum.
Summary updated