Rich Feature Combination for Cost-Based Broad Learning System

Tian-Lun Zhang, Rong Chen, Xi Yang, Shikai Guo

Abstract

Broad learning system (BLS) is an emerging learning algorithm for connectionist models, gaining popularity due to its efficiency and flexibility. As an alternative to deep structures, BLS develops an incremental neural network that is modeled non-iteratively. This paper proposes two major enhancements: (1) a Broadly-Fused Feature (BFR) representation using unsupervised group-wise encoding and lateral feature fusion to improve multi-level feature learning, and (2) a cost-sensitive BLS framework (CR-BLS) to address class imbalance. Extensive experiments on computer vision and bug report datasets demonstrate the effectiveness of the proposed framework.

Introduction

Deep learning has revolutionized artificial intelligence but often suffers from high computational costs and difficulty in incremental updates. The Broad Learning System (BLS) offers a fast, non-iterative alternative by expanding "wide" rather than "deep," using the Moore-Penrose generalized inverse for rapid training. However, standard BLS typically relies on random feature mapping and assumes equal misclassification costs, which can limit its performance on complex or imbalanced datasets. This research introduces R-BLS (Rich Broad Learning System) and its cost-sensitive variant to overcome these limitations.

Methodology

Broadly-Fused Feature (BFR) Representation

The R-BLS framework replaces the standard random feature mapping with a more sophisticated multi-level representation designed to capture complex data characteristics.

Unsupervised Group-wise Encoding

Instead of random weights, R-BLS utilizes Sparse Autoencoders (AE) to extract compact, meaningful features from the input data. By applying an $l_1$-norm penalty, the model learns sparse representations that capture the underlying structure of the data more effectively than random projections.

Structure of the sparse autoencoder used for feature extraction.

Lateral Feature Fusion

To mimic the hierarchical nature of deep networks within a broad architecture, R-BLS implements lateral connections between feature groups. The output of one feature group serves as the input for the next, allowing the model to "reuse" and fuse features across the broad layer. This structure enables the learning of semantically stronger representations without the computational overhead of deep backpropagation.

Architecture of the R-BLS framework showing lateral feature fusion.

Cost-Sensitive Learning (CR-BLS)

To handle imbalanced datasets, where minority classes (e.g., critical software bugs) are often overlooked, the authors introduce a cost-sensitive objective function. By assigning higher weights ($k_{min}$) to minority class misclassifications, the model minimizes the total cost of errors rather than just the raw error rate. This approach modifies the optimization objective directly, avoiding the risks of overfitting or data loss associated with sampling methods like SMOTE or undersampling.

Experimental Evaluation

The framework was evaluated on a wide range of benchmarks, including UCI datasets, computer vision tasks (MNIST, ORL, Fashion-MNIST), and real-world bug reports from major open-source projects like Eclipse, GNOME, and Mozilla.

  • MNIST Accuracy: R-BLS achieved 98.95% accuracy, outperforming standard BLS (98.74%) and several deep models, including Deep Belief Networks (DBN) and Stacked Autoencoders (SAE).
  • Face Recognition (ORL): R-BLS demonstrated superior performance with limited training data, achieving 91% accuracy with only 5 images per person, consistently beating CNNs and SVMs in this regime.
  • Bug Triaging: In highly imbalanced bug report tasks, the cost-sensitive version (CR-BLS) significantly outperformed standard BLS and other imbalanced learning techniques in both overall accuracy and F-measure, proving its utility for "needle-in-a-haystack" classification problems.

Conclusion

The proposed R-BLS framework demonstrates that "rich" feature engineering in a wide architecture can rival deep models in performance while maintaining the computational efficiency of non-iterative training. By integrating lateral feature fusion and cost-sensitivity, the model provides a robust solution for complex and imbalanced real-world classification tasks. Future work will explore different encoder types and extending the architecture to temporal and sequential data.

Summary updated

Abstract

Broad learning system (BLS) is an emerging learning algorithm for the connectionist models, which have enjoyed much popularity on many applications. As an alternative approach of learning in deep structure, the BLS develops an incremental learning neural network that can be modeled in a flexible way, and becomes a promising technique in the field of knowledge discovery and data engineering. To further improve the performance of BLS, our focus is to investigate these algorithms which can enhance the BLS. On one hand, from the viewpoint of feature engineering, unsupervised group-wise encoding is conducted for feature extraction, and broadly fused feature representation is used to improve the ability of BLS, in terms of the learning and reusing multi-level features. On the other hand, for imbalanced learning from disproportionate size of categories instances, a cost-sensitive BLS framework is proposed in this paper, which aims to minimize the total misclassifying cost in classification learning. Finally, we conduct extensive experiments on a wide range of datasets (e.g., computer vision and bug reports) to demonstrate the effectiveness of the proposed BLS framework.
BibTeX
@inproceedings{zhang2018rich,
  author = {Tian-Lun Zhang and Rong Chen and Xi Yang and Shikai Guo},
  title = {{Rich Feature Combination for Cost-Based Broad Learning System}},
  booktitle = {IEEE Access},
  year = {2018},
  doi = {10.1109/access.2018.2885164},
}
Figure