Abstract
Evaluating the Rationales of Amateur Investors (ERAI) is a task focused on mining expert-like viewpoints from social media. This paper summarizes the solutions proposed by the University of Adelaide (UOA) for the ERAI shared task at FinNLP-2022. We address two sub-tasks: a pairwise comparison task using a label-aware BERT-based model and an unsupervised ranking task using regression. Our approach achieved competitive results, placing 2nd overall in the pairwise comparison task.
Introduction
Social media platforms are saturated with financial opinions, making it difficult to distinguish rational, expert-like advice from noise. The ERAI task aims to automate the identification of high-quality rationales based on two metrics: Maximal Potential Profit (MPP) and Maximal Loss (ML). Sub-task 1 involves pairwise comparison of posts to determine which is more rational, while Sub-task 2 requires ranking posts to identify the top 10% with the best financial profiles.
Methodology
Label-Aware Binary Classification (Sub-task 1)
Traditional classification models often ignore the semantic meaning of class labels. We propose a "Label-Aware" approach where natural language descriptions of the labels are appended directly to the input sequence. This allows the model to project both the opinions and the task objectives into a shared semantic space.
The input format follows: [CLS] Post 1 [SEP] Post 2 [SEP] MPP Label Info [SEP] ML Label Info [SEP]. The label info strings are literal descriptions such as "higher maximum possible profit" or "lower maximum loss."
Regression for Ranking (Sub-task 2)
For the ranking task, we fine-tuned Pre-trained Language Models (PLMs) to perform regression. We utilized a dense pooling layer with dropout on the [CLS] token embedding to predict continuous MPP and ML values, optimized using Mean Squared Error (MSE) loss.
Experimental Setup
We evaluated three Chinese-language models: BERT-Base-Chinese, RoBERTa-wwm-ext, and Astock (a BERT model specifically pre-trained on financial news). The ERAI dataset provided 200 labeled pairs for training Sub-task 1 and 210 individual posts for Sub-task 2.
Results and Discussion
Our system, designated as UOA_1, demonstrated strong performance across both tasks. In Sub-task 1, we achieved an accuracy of 54.02% for ML and 51.72% for MPP, securing a 2nd place ranking overall in the competition.
Key findings include:
- Semantic Labels: Incorporating label descriptions significantly improved the model's ability to distinguish rationales by providing explicit task intent.
- Domain Specificity: The
Astockmodel outperformed others in the regression task (Sub-task 2), achieving an average MPP of 12.35% and ML of -9.39%. This highlights the importance of domain-specific pre-training for predicting numerical financial metrics. - Rationality as a Proxy: The results suggest that the quality of textual arguments (rationality) is a viable indicator of potential financial performance.
Conclusion
We presented a robust framework for financial opinion mining that leverages semantic label information and domain-specific PLMs. Future work will explore multi-layer feature extraction and data augmentation techniques to further enhance performance on small-scale financial datasets.
Summary updated