Maximal Uncorrelated Multinomial Logistic Regression

Dajiang Lei, Hongyu Zhang, Hongtao Liu, Zhixing Li, Yu WuIEEE Access 2019 · DOI

Abstract

Multinomial logistic regression (MLR) has been widely used in the field of face recognition, text classification, and so on. However, the standard multinomial logistic regression has not yet stressed the problem of data redundancy. That is to say, in multi-class classification, there are many similar features among different classes, which will cause the corresponding classes not being correctly classified. As data redundancy is a common phenomenon in many fields, in response to this phenomenon, this paper proposes a maximal uncorrelated MLR (MUMLR) classification model to solve the problem of data redundancy in multi-class classification. The main idea is to reduce the weight of similar features and try to keep more discriminative information in the data by adding an uncorrelated regularization. In addition, we use the Cauchy–Buniakowsky–Schwarz inequation to scale the original objective function into the convex function and solve it by the Adam optimization method. Its main advantages are as follows: for data with more redundant information, the classification effect of the proposed algorithm is better than the state-of-the-art algorithms. In addition, we prove that the regularization we proposed can also be applied to neural networks and has achieved good results.

Introduction

In the realm of multi-class classification, data redundancy presents a significant challenge. This redundancy occurs when different categories share highly similar features—for instance, in facial recognition where multiple subjects are photographed against an identical background, or in handwriting recognition where digits like '1' and '7' share nearly identical vertical strokes. Standard Multinomial Logistic Regression (MLR) models often fail to account for this, leading to correlated weight vectors across classes. This "blurring" of class boundaries reduces the model's ability to distinguish between similar categories, ultimately degrading classification accuracy.

While traditional regularization techniques like Lasso ($L_1$) and Ridge ($L_2$) address noise and sparsity, they do not explicitly target the inter-class parameter correlation caused by redundant data. To bridge this gap, we introduce Maximal Uncorrelated Multinomial Logistic Regression (MUMLR).

Methodology

The core innovation of MUMLR is the introduction of an uncorrelated regularization term into the standard MLR cost function.

Uncorrelated Regularization

The fundamental premise is that if two classes are highly similar, their respective weight vectors ($\theta_i$ and $\theta_j$) will exhibit high similarity, resulting in a large dot product ($\theta_i^T \theta_j$). By adding a penalty term that minimizes these cross-class correlations, the model is forced to prioritize features that are unique to each class rather than those shared across the dataset.

Mathematical Optimization via CBS Inequality

Directly optimizing a raw correlation term is computationally intensive due to differentiability issues. To resolve this, we employ the Cauchy–Buniakowsky–Schwarz (CBS) inequality. This mathematical transformation allows us to scale the complex correlation penalty into a simpler product of squared norms ($||\theta_i||^2 ||\theta_j||^2$). This results in a quasi-convex objective function that can be efficiently solved using the Adam optimization method.

Extension to Deep Learning

A significant advantage of the MUMLR framework is its versatility. We demonstrate that this uncorrelated regularization can be seamlessly integrated into the final "softmax" layer of Deep Neural Networks (DNNs) and Convolutional Neural Networks (CNNs). This "plug-and-play" capability allows modern architectures to inherit the robustness of MUMLR against redundant features with negligible computational overhead.

Experimental Evaluation

We evaluated MUMLR against state-of-the-art variants, including Sparse MLR and Weight-Decay MLR, across several benchmarks:

  1. Synthetic Datasets: Using Gaussian, Laplace, and Mixed distributions, we simulated "among-class similarity" levels from 0.5 to 0.9.
  2. MNIST: The standard benchmark for handwritten digit recognition.
  3. COIL20 & ORL: Datasets focusing on object and face recognition, respectively.
  4. Global Temperature (GT): A real-world dataset involving sea surface temperatures.

Key Results

  • Performance in High-Similarity Scenarios: On synthetic data with a similarity index of 0.9, MUMLR achieved approximately 57% accuracy, significantly outperforming competing algorithms which plateaued around 50-51%.
  • The '1 vs 7' Problem: In the MNIST dataset, the confusion between digits 1 and 7 is a common failure mode. MUMLR reduced the misclassification rate for these specific digits to 0.32%, compared to 0.83% for Sparse MLR and 0.90% for Weight-Decay MLR.
  • Benchmark Superiority: MUMLR achieved top-tier recognition rates across MNIST (87.25%), COIL20 (99.58%), and ORL (96.52%).
  • Convergence Efficiency: Experimental data indicates that MUMLR converges faster than its predecessors, making it a practical choice for large-scale datasets.

Conclusion and Future Work

MUMLR provides a robust solution to the problem of data redundancy by forcing models to identify unique, discriminative class features. By penalizing parameter correlation, the model achieves better generalization and higher accuracy in challenging classification tasks.

The current implementation faces a quadratic computational complexity ($O(k^2)$) relative to the number of classes ($k$), which may pose challenges for massive datasets like ImageNet. Future research will focus on developing approximation algorithms to maintain performance while reducing the computational burden for high-dimensional class spaces.

Summary updated

Abstract

Multinomial logistic regression (MLR) has been widely used in the field of face recognition, text classification, and so on. However, the standard multinomial logistic regression has not yet stressed the problem of data redundancy. That is to say, in multi-class classification, there are many similar features among different classes, which will cause the corresponding classes not being correctly classified. As data redundancy is a common phenomenon in many fields, in response to this phenomenon, this paper proposes a maximal uncorrelated MLR (MUMLR) classification model to solve the problem of data redundancy in multi-class classification. The main idea is to reduce the weight of similar features and try to keep more discriminative information in the data by adding an uncorrelated regularization. In addition, we use the Cauchy–Buniakowsky–Schwarz inequation to scale the original objective function into the convex function and solve it by the Adam optimization method. Its main advantages are as follows: for data with more redundant information, the classification effect of the proposed algorithm is better than the state-of-the-art algorithms. In addition, we prove that the regularization we proposed can also be applied to neural networks and has achieved good results.
BibTeX
@inproceedings{lei2019maximal,
  author = {Dajiang Lei and Hongyu Zhang and Hongtao Liu and Zhixing Li and Yu Wu},
  title = {{Maximal Uncorrelated Multinomial Logistic Regression}},
  booktitle = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2019.2921820},
}
Figure