TL;DR
This paper proposes a novel approach to train Random Vector Functional Link (RVFL) networks for edge devices by replacing the computationally expensive least-squares classifier with Generalized Learning Vector Quantization (GLVQ). This modification significantly reduces training costs while achieving competitive or superior classification accuracy, making RVFL networks more practical for resource-constrained environments.
What Problem Does This Paper Address?
Machine learning algorithms are increasingly being deployed on edge devices, which have strict limitations on computational resources and energy consumption. Random Vector Functional Link (RVFL) networks are attractive for these applications due to their simple architecture and efficient training. However, the standard RVFL network relies on least-squares regression for training, which involves computationally intensive matrix operations like inversion. These operations are particularly problematic for specialized hardware like FPGAs commonly found in edge devices. The paper aims to address this by finding a more efficient and hardware-friendly alternative to the least-squares classifier for RVFL networks without sacrificing accuracy.
Methods
The core contribution of this paper is the replacement of the least-squares (LS) classifier in RVFL networks with the Generalized Learning Vector Quantization (GLVQ) algorithm.
- RVFL Networks: These are shallow neural networks with a single hidden layer. The connections between the input and hidden layers are fixed and randomly initialized. Only the connections between the hidden and output layers (readout weights) are trained.
- Standard RVFL Training (Least-Squares): Traditionally, the readout weights are trained by solving a linear least-squares problem, often using matrix inversion. This is computationally expensive and can be numerically unstable.
- Proposed Approach (RVFL + GLVQ): The paper replaces the least-squares classifier with GLVQ. GLVQ is an iterative learning algorithm that aims to find prototypes (representatives) for each class. Instead of matrix operations, GLVQ relies on simple vector and distance calculations, making it more suitable for hardware implementation. The hidden layer activations of the RVFL network serve as the input to the GLVQ classifier.
- Hyperdimensional Computing (HDC) Integration: The paper also leverages ideas from HDC, particularly in the context of "intRVFL" networks, which use integer-only activations and hypervectors. The proposed approach can be seen as an improvement over standard HDC classification methods that often use simple centroid-based approaches.
Datasets
The experiments were conducted on a collection of 121 real-world classification datasets from the UCI Machine Learning Repository. No new benchmarks were introduced; the paper followed the experimental protocol described in a seminal work [13] that used this collection. Input features were normalized to the range [0, 1].
Key Insights and Ideas
- GLVQ as a Viable Alternative to Least-Squares: The paper demonstrates that GLVQ can effectively replace least-squares for training RVFL networks, offering a significant computational advantage without a substantial drop in accuracy.
- Efficiency Gains on Edge Devices: By avoiding matrix operations, the GLVQ-based RVFL approach is much more amenable to hardware implementation on resource-constrained edge devices.
- Flexibility in Training Iterations: GLVQ's iterative nature allows for flexible control over training time. The paper shows that even with a severely limited number of iterations (e.g., 10), competitive accuracy can be maintained, leading to substantial computational cost savings.
- Improved Classification over Standard HDC: GLVQ, especially with multiple prototypes per class, offers a more sophisticated approach to classification compared to simple centroid-based methods often used in HDC, leading to better accuracy by capturing intra-class variability.
Key Results
- Accuracy: The proposed RVFL+GLVQ approach achieved state-of-the-art accuracy on the UCI datasets, outperforming previously proposed RVFL networks.
- When using one prototype per class, GLVQ achieved an average accuracy of 0.76, comparable to the RLS classifier (0.73).
- With multiple prototypes per class, the average accuracy increased to 0.82, outperforming the RLS classifier (0.80).
- Computational Cost: The GLVQ classifier, when limited to a fixed number of iterations, used on average only 21% of the computational cost of the least-squares classifier. This is a significant reduction for training on edge devices.
- Comparison with intRVFL: The proposed approach also showed competitive results against intRVFL networks, with GLVQ-based intRVFL achieving an average accuracy of 0.81 compared to RLS-based intRVFL at 0.80.
Limitations and Future Work
- Optimal Number of Prototypes: The paper acknowledges that the optimal number of prototypes per class is a hyperparameter that affects generalization error. While they explored multiple prototypes, they did not investigate assigning variable numbers of prototypes to different classes, which could further optimize performance.
- Computational Cost vs. Iterations: The paper notes that while GLVQ is computationally cheaper than RLS, its total cost is dependent on the number of iterations. However, they demonstrate that limiting iterations significantly reduces cost with minimal accuracy loss.
- Future Work: The paper suggests that the proposed approach can flexibly improve the accuracy and efficiency of machine learning applications when hardware resources are limited. Further exploration of hyperparameter tuning, particularly the number of prototypes per class, and its impact on generalization error, would be valuable.
Summary updated