Yet at the FinNLP-2022 ERAI Task: Modified models for evaluating the Rationales of Amateur Investors

Yan Zhuang, Fuji Ren2022 · DOI

Abstract

The financial reports usually reveal the recent development of the company and often cause the volatility in the company's share price. The opinions causing higher maximal potential profit and lower maximal loss can help the amateur investors choose rational strategies. FinNLP-2022 ERAI task aims to quantify the opinions' potentials of leading higher maximal potential profit and lower maximal loss. In this paper, different strategies were applied to solve the ERAI tasks. Valinna 'RoBERTa-wwm' showed excellent performance and helped us rank second in 'MPP' label prediction task. After integrating some tricks, the modified 'RoBERTa-wwm' outperformed all other models in 'ML' ranking task.

Introduction

Social media platforms are increasingly flooded with investment "rationales" from amateur investors. While some advice is valuable, much of it is noise. The Evaluating the Rationales of Amateur Investors (ERAI) task at FinNLP-2022 aims to quantify the quality of these opinions by predicting two key metrics: Maximal Potential Profit (MPP), representing the highest possible return, and Maximal Loss (ML), representing the greatest downside risk. This task is challenging due to the informal, multilingual nature of social media posts, which require models to understand both general sentiment and specific financial nuances.

Methods

The researchers explored several BERT-style architectures, including domain-specific models like FinBERT and Mengzi-Fin. The most successful approach utilized a "Modified-RoBERTa-wwm" (Whole Word Masking) model. To optimize performance, three key techniques were integrated:

  1. Stochastic Weight Averaging (SWA): Averaging weights from different training stages to improve generalization and find flatter optima.
  2. MADGRAD Optimizer: A momentumized, adaptive gradient method that replaced the standard AdamW optimizer.
  3. Multi-sample Dropout: Using multiple dropout masks on the final hidden state ([CLS] token) to reduce variance and prevent overfitting.

The ERAI dataset, sourced from Taiwanese platforms PTT Stock and Mobile01, includes posts in both English and Chinese. Preprocessing involved converting Traditional Chinese to Simplified Chinese to better align with pre-trained models and removing noise like URLs and emoticons. The tasks included Pairwise Comparison (sentence-pair classification) and Unsupervised Ranking (regression for the top 10% of posts).

Results and Analysis

The team's strategies proved highly competitive. In the MPP classification task, the vanilla RoBERTa-wwm model achieved an offline accuracy of 63.98%, securing 2nd place in the official competition. Interestingly, general-purpose models often outperformed finance-specific ones, suggesting that amateur investor language aligns more closely with general web text than formal financial reports.

Performance comparison of various models across MPP and ML tasks.

In the ranking task, the "Modified" version of the model (incorporating SWA and MADGRAD) was significantly more effective. It ranked 1st in the competition for predicting the top 10% of posts with the lowest maximal loss. However, as shown in the ranking results, while the modified model performed well, it still struggled to surpass the provided baseline in the MPP ranking task (14.61% vs. 17.61%), indicating that predicting high-profit "alpha" from text remains a difficult problem.

Ranking performance for the top 10% of posts, highlighting the Modified-RoBERTa-wwm model.

Conclusion

The study demonstrates that optimization "tricks" like SWA and MADGRAD can significantly enhance the performance of general-purpose models for specialized financial tasks. Future work will focus on modeling the negative correlation between MPP and ML and conducting detailed ablation studies to isolate the impact of specific optimization techniques on model performance.

Summary updated

Abstract

The financial reports usually reveal the recent development of the company and often cause the volatility in the company's share price.The opinions causing higher maximal potential profit and lower maximal loss can help the amateur investors choose rational strategies.FinNLP-2022 ERAI task aims to quantify the opinions' potentials of leading higher maximal potential profit and lower maximal loss.In this paper, different strategies were applied to solve the ERAI tasks.Valinna 'RoBERTa-wwm' showed excellent performance and helped us rank second in 'MPP' label prediction task.After integrating some tricks, the modified 'RoBERTa-wwm' outperformed all other models in 'ML' ranking task.
BibTeX
@misc{zhuang2022yet,
  author = {Yan Zhuang and Fuji Ren},
  title = {{Yet at the FinNLP-2022 ERAI Task: Modified models for evaluating the Rationales of Amateur Investors}},
  year = {2022},
  doi = {10.18653/v1/2022.finnlp-1.17},
}
Figure