Sentiment Analysis About Investors and Consumers in Energy Market Based on BERT-BiLSTM

Ren Cai, Bin Qin, Yangken Chen, Liang Zhang, Ruijiang Yang, Shiwei Chen, Wei WangIEEE Access 2020 · DOI

Introduction

Traditional financial theories often struggle to account for the extreme volatility observed in energy markets. While supply and demand are fundamental drivers, human emotion—ranging from investor panic to consumer optimism—plays a significant role, particularly during "black swan" events like the COVID-19 pandemic. Analyzing this sentiment is challenging because social media and online forums are flooded with messy, unstructured data that varies wildly in style and quality.

Standard sentiment analysis tools frequently fail to capture the nuances of Chinese-language comments or the long-term dependencies in text that signal a user's true market outlook. To address this, researchers have developed a hybrid deep learning model, BERT-BiLSTM, designed to provide robust technical support for market forecasting by effectively "reading the room" of the energy market.

Methodology

The proposed model utilizes a "combination forecasting method" that stacks two powerful architectures to handle the complexities of natural language.

BERT for Contextual Embeddings

The upstream component of the model is BERT (Bidirectional Encoder Representations from Transformer). Unlike traditional models like word2vec that assign a fixed vector to a word regardless of context, BERT generates dynamic representations based on surrounding words. The researchers employed a simplified Chinese training model with 12 layers and 110 million parameters to extract high-quality word vectors that capture the semantic essence of investor comments.

BERT Pre-training and Fine-tuning Architecture

BiLSTM for Sequence Modeling

The downstream component is BiLSTM (Bidirectional Long Short Term Memory). While BERT excels at understanding context, BiLSTM is specialized in processing sequences. By analyzing text both forward and backward, it captures long-term dependencies and the grammatical logic of human language better than standard Recurrent Neural Networks (RNNs).

Internal Structure of the LSTM Unit

In this hybrid approach, the BERT output is fed directly into the BiLSTM layer, followed by a Softmax layer for final sentiment classification. This synergy allows the model to cover both the "micro" statistical characteristics of nearby words and the "macro" logic of the entire sentence.

Data and Experimental Setup

The study utilized a dataset of approximately 100,000 comments from investors and consumers published on the Chinese Internet between late 2019 and the first half of 2020.

  • Preprocessing: Data was cleaned to remove invalid entries such as empty text or punctuation-only strings.
  • Balancing: To address a natural imbalance (a 1:5 ratio of positive to negative/neutral comments), the researchers used upsampling and downsampling to achieve a 1:2 ratio for the training set.
  • Split: The dataset was divided into training, validation, and test sets using a 7:1:2 ratio.

Results and Discussion

The hybrid BERT-BiLSTM model demonstrated superior performance across all key metrics when compared to standalone BERT and LSTM models.

Model Accuracy Recall (Positive) F1 Score
BERT-BiLSTM 0.8620 0.7078 0.6357
BERT 0.8559 0.5576 0.5684
BiLSTM 0.7775 0.0747 0.1025

A critical finding was the "Recall Gap." While standalone BERT achieved decent accuracy, its recall for positive sentiment (Label 1) was only 0.5576. The addition of the BiLSTM layer boosted this to 0.7078, indicating that sequence modeling is essential for identifying specific sentiment orientations in complex financial discourse. Conversely, the standalone BiLSTM failed significantly (recall of 0.0747), proving that sequence models require high-quality feature extraction from BERT to handle raw social media text.

Conclusion

The BERT-BiLSTM model offers a more accurate way to monitor sentiment in the energy market, providing valuable insights for investors and policymakers. Future work will focus on scaling the model with larger datasets and leveraging more powerful computing clusters to refine predictive accuracy for real-time decision-making.

Summary updated

BibTeX
@article{cai2020sentiment,
  author = {Ren Cai and Bin Qin and Yangken Chen and Liang Zhang and Ruijiang Yang and Shiwei Chen and Wei Wang},
  title = {{Sentiment Analysis About Investors and Consumers in Energy Market Based on BERT-BiLSTM}},
  journal = {IEEE Access},
  year = {2020},
  doi = {10.1109/access.2020.3024750},
}
Figure