UOA at the FinNLP-2022 ERAI Task: Leveraging the Class Label Description for Financial Opinion Mining

Jinan Zou, Haiyao Cao, Yanxi Liu, Lingqiao Liu, Ehsan Abbasnejad, Qinfeng Shi2022 · DOI

Abstract

Evaluating the Rationales of Amateur Investors (ERAI) is a task focused on mining expert-like viewpoints from social media. This paper summarizes the solutions proposed by the University of Adelaide (UOA) for the ERAI shared task at FinNLP-2022. We address two sub-tasks: a pairwise comparison task using a label-aware BERT-based model and an unsupervised ranking task using regression. Our approach achieved competitive results, placing 2nd overall in the pairwise comparison task.

Introduction

Social media platforms are saturated with financial opinions, making it difficult to distinguish rational, expert-like advice from noise. The ERAI task aims to automate the identification of high-quality rationales based on two metrics: Maximal Potential Profit (MPP) and Maximal Loss (ML). Sub-task 1 involves pairwise comparison of posts to determine which is more rational, while Sub-task 2 requires ranking posts to identify the top 10% with the best financial profiles.

Methodology

Label-Aware Binary Classification (Sub-task 1)

Traditional classification models often ignore the semantic meaning of class labels. We propose a "Label-Aware" approach where natural language descriptions of the labels are appended directly to the input sequence. This allows the model to project both the opinions and the task objectives into a shared semantic space.

Architecture of the label-aware BERT model for pairwise comparison.

The input format follows: [CLS] Post 1 [SEP] Post 2 [SEP] MPP Label Info [SEP] ML Label Info [SEP]. The label info strings are literal descriptions such as "higher maximum possible profit" or "lower maximum loss."

Regression for Ranking (Sub-task 2)

For the ranking task, we fine-tuned Pre-trained Language Models (PLMs) to perform regression. We utilized a dense pooling layer with dropout on the [CLS] token embedding to predict continuous MPP and ML values, optimized using Mean Squared Error (MSE) loss.

Experimental Setup

We evaluated three Chinese-language models: BERT-Base-Chinese, RoBERTa-wwm-ext, and Astock (a BERT model specifically pre-trained on financial news). The ERAI dataset provided 200 labeled pairs for training Sub-task 1 and 210 individual posts for Sub-task 2.

Results and Discussion

Our system, designated as UOA_1, demonstrated strong performance across both tasks. In Sub-task 1, we achieved an accuracy of 54.02% for ML and 51.72% for MPP, securing a 2nd place ranking overall in the competition.

Final accuracy rankings for Sub-task 1 across all participating models.

Key findings include:

  • Semantic Labels: Incorporating label descriptions significantly improved the model's ability to distinguish rationales by providing explicit task intent.
  • Domain Specificity: The Astock model outperformed others in the regression task (Sub-task 2), achieving an average MPP of 12.35% and ML of -9.39%. This highlights the importance of domain-specific pre-training for predicting numerical financial metrics.
  • Rationality as a Proxy: The results suggest that the quality of textual arguments (rationality) is a viable indicator of potential financial performance.

Conclusion

We presented a robust framework for financial opinion mining that leverages semantic label information and domain-specific PLMs. Future work will explore multi-layer feature extraction and data augmentation techniques to further enhance performance on small-scale financial datasets.

Summary updated

Abstract

Evaluating the Rationales of Amateur Investors (ERAI) is a task about mining expert-like viewpoints from social media. This paper summarizes our solutions to the ERAI shared task, which is co-located with the FinNLP workshop at EMNLP 2022. There are 2 sub-tasks in ERAI. Sub-task 1 is a pair-wised comparison task, where we propose a BERT-based pre-trained model projecting opinion pairs in a common space for classification. Sub-task 2 is an unsupervised learning task ranking the opinions’ maximal potential profit (MPP) and maximal loss (ML), where our model leverages the regression method and multi-layer perceptron to rank the MPP and ML values. The proposed approaches achieve competitive accuracy of 54.02% on ML Accuracy and 51.72% on MPP Accuracy for pairwise tasks, also 12.35% and -9.39% regression unsupervised ranking task for MPP and ML.
BibTeX
@misc{zou2022uoa,
  author = {Jinan Zou and Haiyao Cao and Yanxi Liu and Lingqiao Liu and Ehsan Abbasnejad and Qinfeng Shi},
  title = {{UOA at the FinNLP-2022 ERAI Task: Leveraging the Class Label Description for Financial Opinion Mining}},
  year = {2022},
  doi = {10.18653/v1/2022.finnlp-1.15},
}
Figure