Enhancing Sentiment Classification with Machine Learning and Combinatorial Fusion

Sean Patten, Pin-Yu Chen, Christina Schweikert, D. Frank HsuarXiv 2025 · DOI · arXiv

Abstract

Sentiment analysis remains a critical challenge in Natural Language Processing (NLP) due to the inherent complexities of human language, such as sarcasm and irony. While the prevailing trend in the field focuses on scaling Transformer-based models, this research explores an alternative path: Combinatorial Fusion Analysis (CFA). By intelligently merging a fine-tuned RoBERTa transformer with traditional machine learning models, researchers achieved a new state-of-the-art accuracy of 97.072% on the IMDb benchmark, demonstrating that cognitive diversity between models can be more effective than raw scale.

Introduction

The "bias-variance dilemma" often plagues sentiment classification; complex models may overfit specific patterns, while simpler models may fail to capture nuanced linguistic features. Traditional ensemble methods, such as simple averaging, often overlook the unique "perspectives" that heterogeneous models bring to a dataset. This study addresses this gap by applying CFA to exploit the unique strengths of diverse architectures, moving beyond simple performance-weighted combinations to focus on the informational diversity of the ensemble members.

Methodology

The research team constructed an ensemble consisting of four distinct base models to ensure high feature-level diversity:

  1. RoBERTa (Model A): A state-of-the-art transformer utilizing deep subword embeddings.
  2. SVM (Model B): A classical linear classifier.
  3. XGBoost (Model C): A gradient boosting framework.
  4. Random Forest (Model D): A traditional ensemble of 25,000 decision trees.

Combinatorial Fusion Analysis (CFA)

The core innovation lies in the use of CFA to quantify and leverage "Cognitive Diversity" (CD). The process involves mapping each model's output using a Rank-Score Characteristic (RSC) Function, which visualizes how a model assigns confidence scores relative to its internal rankings.

The Rank-Score Characteristic (RSC) function illustrates the distinct confidence profiles of the four base models, highlighting the high-confidence sigmoidal curve of RoBERTa versus the more distributed curves of classical models.

By measuring the difference in how two models rank the same data, the researchers calculated the CD metric. This allowed for the implementation of Weighted Combination by Diversity Strength (WCDS), where models are weighted based on the unique information they contribute rather than their individual accuracy alone.

Key Results and Discussion

The application of diversity-weighted fusion yielded significant performance gains over both individual models and standard ensemble techniques. The ensemble achieved a record-breaking accuracy of 97.072%, surpassing the previous state-of-the-art for the IMDb dataset.

The experimental results confirm that the Diversity-based Weighted Score Combination (WCDS) achieved the highest accuracy of 97.072% using the (A, B, D) model combination.

Insights on Diversity

  • Complementary Biases: Analysis revealed that RoBERTa exhibited a slight bias toward positive predictions (higher recall), while the SVM was more conservative (higher precision). The CFA fusion moderated these biases, resulting in a more balanced and robust classifier.
  • Efficiency: Once base models are trained, the CFA fusion step is computationally negligible, requiring only simple arithmetic. This provides a scalable path to high-performance NLP without the necessity for massive GPU clusters.
  • Diversity vs. Performance: WCDS consistently outperformed performance-weighted fusion (WCP). This confirms that a collection of models that make different types of errors is more valuable than a collection of the highest-performing models that fail in the same way.

Conclusion

This research demonstrates that the intelligent fusion of heterogeneous models through Combinatorial Fusion Analysis can significantly outperform individual state-of-the-art transformers. By prioritizing cognitive diversity, the proposed method achieved a new benchmark on the IMDb dataset while maintaining computational efficiency. Future work will explore the application of this framework to more diverse datasets, such as social media and financial news, and investigate the inclusion of newer architectures like XLNet to further push the boundaries of cognitive diversity in sentiment classification.

Summary updated

BibTeX
@misc{patten2025enhancing,
  author = {Sean Patten and Pin-Yu Chen and Christina Schweikert and D. Frank Hsu},
  title = {{Enhancing Sentiment Classification with Machine Learning and Combinatorial Fusion}},
  year = {2025},
  eprint = {2510.27014},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.27014},
}
Figure