Abstract
This paper provides an overview of the shared task, Evaluating the Rationales of Amateur Investors (ERAI), in FinNLP-2022 at EMNLP-2022. This shared task aims to sort out investment opinions that would lead to higher profit from social platforms. We obtained 19 registered teams; 9 teams submitted their results for final evaluation, and 8 teams submitted papers to share their methods. The discussed directions are various: prompting, fine-tuning, translation system comparison, and tailor-made neural network architectures. We provide details of the task settings, data statistics, participants’ results, and fine-grained analysis.
Introduction
Social media platforms like Reddit, Mobile01, and PTT are flooded with thousands of investment opinions daily. For an investor, the challenge isn't finding information; it's filtering out the low-quality "noise" to find opinions backed by rationales that actually lead to profit.
Traditional sentiment analysis, which merely identifies if a post is "bullish" or "bearish," is insufficient for evaluating the quality of financial advice. An investor needs to know if the reasoning behind that sentiment is sound. To address this, the ERAI task proposes using historical market data as an objective proxy for quality. If an opinion's rationale leads to a high Maximum Possible Profit (MPP) and low Maximum Loss (ML), it is deemed high quality. The ERAI task aims to automate this evaluation process.
Task Definition and Datasets
The shared task was split into two distinct challenges:
- Pairwise Comparison: Given two investment opinions, models must predict which one will lead to a higher MPP and which will result in a greater ML. This is framed as a binary classification task.
- Unsupervised Ranking: Given a pool of opinions, models must rank them without using profit/loss labels during training. The goal is to identify the top 10% of posts that lead to the highest MPP.
The researchers curated two primary datasets, originally in Chinese and translated to English:
- Mobile01 Dataset: Used for the pairwise task, consisting of 574 posts (287 pairs).
- PTT Dataset: Used for the unsupervised ranking task, containing 210 posts.
- Labels: Every post was manually checked for sentiment and then labeled with calculated MPP and ML values based on actual market price fluctuations following the post's date.
Methodologies
Participants employed a wide range of NLP techniques to tackle these challenges. These included prompting Large Language Models (LLMs) like GPT-3 and T5, fine-tuning domain-specific models like FinBERT, and utilizing specialized financial lexicons (e.g., NTUSD-Fin, Loughran-McDonald) to score the "expert-likeness" of the text. Some teams also developed custom architectures like AStock and DPP-VAE.
Key Results and Analysis
The evaluation revealed several critical insights into the nature of financial social media analysis:
- General vs. Domain-Specific: Surprisingly, general-purpose language models (like BERT-Chinese) often outperformed domain-specific models (like FinBERT) in the MPP comparison task. This suggests that the "rationality" of an amateur investor might be expressed in common language rather than dense financial jargon.
- Sentiment vs. Risk: Positive/negative sentiment is a strong indicator for potential profit (MPP), but it is a poor predictor of risk (ML). Predicting how much an investment might lose requires a different analytical approach than predicting how much it might gain.
- The "Expert" Proxy: In the unsupervised setting, simple lexicon-based methods that count "expert-like" sentences were remarkably effective, highlighting that professional-sounding language is a strong signal for lower-risk opinions.
In terms of quantitative performance, the top-performing model for Pairwise MPP (Jetsons_1) achieved an accuracy of 62.07%, while the best model for predicting loss (DCU-ML_1) reached 59.77%. In the unsupervised ranking task, the best approach (PromptShots_2) achieved a top-10% average MPP of 24.39%, significantly outperforming the baseline of 17.61%.
Conclusion and Future Work
The ERAI task highlights the difficulty of separating signal from noise in amateur financial discourse. Future research will need to address the limited context windows of current models to better process long-form rationales and improve data preprocessing to filter out non-informative posts. Furthermore, the authors plan to explore argumentation theory—specifically identifying the structure of claims and premises within an investor's post—to better score their logic and move beyond simple sentiment metrics.
Summary updated