Overview of the FinNLP-2022 ERAI Task: Evaluating the Rationales of Amateur Investors

Chung-Chi Chen, Hen‐Hsen Huang, Hiroya Takamura, Hsin‐Hsi Chen2022 · DOI

Abstract

This paper provides an overview of the shared task, Evaluating the Rationales of Amateur Investors (ERAI), in FinNLP-2022 at EMNLP-2022. This shared task aims to sort out investment opinions that would lead to higher profit from social platforms. We obtained 19 registered teams; 9 teams submitted their results for final evaluation, and 8 teams submitted papers to share their methods. The discussed directions are various: prompting, fine-tuning, translation system comparison, and tailor-made neural network architectures. We provide details of the task settings, data statistics, participants’ results, and fine-grained analysis.

Introduction

Social media platforms like Reddit, Mobile01, and PTT are flooded with thousands of investment opinions daily. For an investor, the challenge isn't finding information; it's filtering out the low-quality "noise" to find opinions backed by rationales that actually lead to profit.

Traditional sentiment analysis, which merely identifies if a post is "bullish" or "bearish," is insufficient for evaluating the quality of financial advice. An investor needs to know if the reasoning behind that sentiment is sound. To address this, the ERAI task proposes using historical market data as an objective proxy for quality. If an opinion's rationale leads to a high Maximum Possible Profit (MPP) and low Maximum Loss (ML), it is deemed high quality. The ERAI task aims to automate this evaluation process.

Task Definition and Datasets

The shared task was split into two distinct challenges:

  1. Pairwise Comparison: Given two investment opinions, models must predict which one will lead to a higher MPP and which will result in a greater ML. This is framed as a binary classification task.
  2. Unsupervised Ranking: Given a pool of opinions, models must rank them without using profit/loss labels during training. The goal is to identify the top 10% of posts that lead to the highest MPP.

The researchers curated two primary datasets, originally in Chinese and translated to English:

  • Mobile01 Dataset: Used for the pairwise task, consisting of 574 posts (287 pairs).
  • PTT Dataset: Used for the unsupervised ranking task, containing 210 posts.
  • Labels: Every post was manually checked for sentiment and then labeled with calculated MPP and ML values based on actual market price fluctuations following the post's date.

Methodologies

Participants employed a wide range of NLP techniques to tackle these challenges. These included prompting Large Language Models (LLMs) like GPT-3 and T5, fine-tuning domain-specific models like FinBERT, and utilizing specialized financial lexicons (e.g., NTUSD-Fin, Loughran-McDonald) to score the "expert-likeness" of the text. Some teams also developed custom architectures like AStock and DPP-VAE.

Summary of participants' methods, language models, and features used in the ERAI task.

Key Results and Analysis

The evaluation revealed several critical insights into the nature of financial social media analysis:

  • General vs. Domain-Specific: Surprisingly, general-purpose language models (like BERT-Chinese) often outperformed domain-specific models (like FinBERT) in the MPP comparison task. This suggests that the "rationality" of an amateur investor might be expressed in common language rather than dense financial jargon.
  • Sentiment vs. Risk: Positive/negative sentiment is a strong indicator for potential profit (MPP), but it is a poor predictor of risk (ML). Predicting how much an investment might lose requires a different analytical approach than predicting how much it might gain.
  • The "Expert" Proxy: In the unsupervised setting, simple lexicon-based methods that count "expert-like" sentences were remarkably effective, highlighting that professional-sounding language is a strong signal for lower-risk opinions.
Performance of participating teams on the pairwise comparison task for MPP and ML.

In terms of quantitative performance, the top-performing model for Pairwise MPP (Jetsons_1) achieved an accuracy of 62.07%, while the best model for predicting loss (DCU-ML_1) reached 59.77%. In the unsupervised ranking task, the best approach (PromptShots_2) achieved a top-10% average MPP of 24.39%, significantly outperforming the baseline of 17.61%.

Conclusion and Future Work

The ERAI task highlights the difficulty of separating signal from noise in amateur financial discourse. Future research will need to address the limited context windows of current models to better process long-form rationales and improve data preprocessing to filter out non-informative posts. Furthermore, the authors plan to explore argumentation theory—specifically identifying the structure of claims and premises within an investor's post—to better score their logic and move beyond simple sentiment metrics.

Summary updated

Abstract

This paper provides an overview of the shared task, Evaluating the Rationales of Amateur Investors (ERAI), in FinNLP-2022 at EMNLP-2022. This shared task aims to sort out investment opinions that would lead to higher profit from social platforms. We obtained 19 registered teams; 9 teams submitted their results for final evaluation, and 8 teams submitted papers to share their methods. The discussed directions are various: prompting, fine-tuning, translation system comparison, and tailor-made neural network architectures. We provide details of the task settings, data statistics, participants’ results, and fine-grained analysis.
BibTeX
@misc{chen2022overview,
  author = {Chung-Chi Chen and Hen‐Hsen Huang and Hiroya Takamura and Hsin‐Hsi Chen},
  title = {{Overview of the FinNLP-2022 ERAI Task: Evaluating the Rationales of Amateur Investors}},
  year = {2022},
  doi = {10.18653/v1/2022.finnlp-1.11},
}
Figure