Density Encoding Enables Resource-Efficient Randomly Connected Neural Networks

Denis Kleyko, Mansour Kheffache, Edward Paxon Frady, Urban Wiklund, Evgeny Osipov

TL;DR

This paper introduces a novel approach to Random Vector Functional Link (RVFL) networks that significantly boosts resource efficiency for edge devices. By combining density-based encoding from stochastic computing with hyperdimensional computing techniques, they achieve higher accuracy, drastically reduced energy consumption, and faster processing times compared to conventional RVFLs, even when using integer-only operations.

What Problem Does This Paper Address?

Deploying machine learning models on resource-constrained edge devices (like smartphones or IoT sensors) is a major challenge. These devices have limited computational power, memory, and energy budgets. While Random Vector Functional Link (RVFL) networks are attractive due to their simple architecture and fast training, their resource demands can still be prohibitive for many edge applications. This paper aims to make RVFLs significantly more resource-efficient without sacrificing, and even improving, their performance.

Methods

The core innovation lies in a new way to represent input features and compute hidden layer activations. Instead of directly feeding input features to the hidden layer, they are first transformed using a density-based encoding (also known as thermometric encoding). This method represents a scalar value by setting a certain number of bits in a vector to "one" (or -1 in the bipolar case) and the rest to "zero" (or +1). This creates a density-based representation layer.

The connections from this representation layer to the hidden layer are structured and based on hyperdimensional computing principles, specifically using binding (position-wise multiplication) and bundling (position-wise addition with clipping). Crucially, the paper proposes using an integer-only readout matrix, further reducing computational complexity and memory footprint. The activation function in the hidden layer is also changed to a clipping function, which is more amenable to integer arithmetic.

Datasets

The proposed approach was evaluated on 121 real-world classification datasets from the UCI Machine Learning Repository. No new benchmarks were introduced; the focus was on evaluating the proposed method against existing ones on a broad set of standard datasets.

Key Insights and Ideas

  • Density-based encoding as a feature transformer: This technique effectively converts real-valued inputs into a structured, distributed representation that is more amenable to efficient computation.
  • Hyperdimensional computing for structured connections: Using binding and bundling operations provides a principled way to create structured, fixed random connections, simplifying the network architecture and enabling integer-only operations.
  • Integer-only arithmetic is feasible and beneficial: The proposed method demonstrates that high accuracy can be maintained even when all computations, including the readout matrix, are performed using integers, leading to significant resource savings.
  • Density-based encoding as implicit regularization: The authors hypothesize that the quantization inherent in the density-based encoding might provide a form of regularization, contributing to the observed accuracy improvements.

Key Results

  • Improved Accuracy: The proposed approach achieved higher average accuracy than the conventional RVFL across 121 datasets.
  • Significant Speedup and Energy Reduction: FPGA implementations of the proposed approach were approximately 11 times more energy-efficient and 2.5 times faster than conventional RVFLs.
  • Robustness to Quantization: Even when the readout matrix weights were quantized to a limited range (e.g., 5-bits), the proposed approach maintained competitive accuracy, outperforming a finite-precision RVFL under a fixed energy budget.

Limitations and Future Work

  • Theoretical Characterization: The authors acknowledge the need for a theoretical characterization of the classification performance improvement, particularly understanding the role of quantization and density-based encoding as regularization.
  • Generalization to Other Architectures: While the focus is on RVFLs, the authors conjecture that the density-based encoding could be beneficial for developing resource-efficient versions of other neural network architectures, such as convolutional neural networks, where binarizing input layers can sometimes harm accuracy.
  • Analytical Theory: Developing an analytical theory similar to capacity theory for these structured representations is suggested as a future research direction, aiming to relate approximation quality to network complexity.

Summary updated

Abstract

The deployment of machine learning algorithms on resource-constrained edge devices is an important challenge from both theoretical and applied points of view. In this article, we focus on resource-efficient randomly connected neural networks known as Random Vector Functional Link (RVFL) networks since their simple design and extremely fast training time make them very attractive for solving many applied classification tasks. We propose to represent input features via the density-based encoding known in the area of stochastic computing and use the operations of binding and bundling from the area of hyperdimensional computing for obtaining the activations of the hidden neurons. Using a collection of 121 real-world datasets from the UCI Machine Learning Repository, we empirically show that the proposed approach demonstrates higher average accuracy than the conventional RVFL. We also demonstrate that it is possible to represent the readout matrix using only integers in a limited range with minimal loss in the accuracy. In this case, the proposed approach operates only on small n-bits integers, which results in a computationally efficient architecture. Finally, through hardware Field-Programmable Gate Array (FPGA) implementations, we show that such an approach consumes approximately eleven times less energy than that of the conventional RVFL.
BibTeX
@article{kleyko2021density,
  author = {Denis Kleyko and Mansour Kheffache and Edward Paxon Frady and Urban Wiklund and Evgeny Osipov},
  title = {{Density Encoding Enables Resource-Efficient Randomly Connected Neural Networks}},
  journal = {IEEE TNNLS},
  year = {2021},
  doi = {10.1109/tnnls.2020.3015971},
}
Figure