In high-stakes environments like finance and politics, outright lies are rare, but "soft misinformation" is rampant. This phenomenon occurs when a respondent provides a factually correct answer that nonetheless obscures the truth by being irrelevant, overly cautious, or intentionally vague.
While the Natural Language Processing (NLP) community has robust tools for objective fact-checking and basic sentiment analysis, there is a significant lack of high-quality, manually annotated data for the nuanced, multi-dimensional features of professional speech. Current financial NLP models often struggle to "read between the lines" of executive responses during the unscripted Q&A sessions of earnings calls.
To bridge this gap, researchers from Georgia Tech have introduced SubjECTive-QA, a new human-annotated dataset designed to quantify the "vibe" of corporate communication. By analyzing Earnings Call Transcripts (ECTs) across six subjective dimensions, the paper provides a framework to detect soft misinformation.
Methodology: Building SubjECTive-QA
The authors developed a rigorous pipeline to move from raw transcripts to a structured dataset capable of training nuanced language models.
- Feature Selection: Initially using the PaLM-2 API to identify prevalent properties in earnings call answers, the researchers leveraged SpaCy for semantic similarity and domain expertise to narrow the list down to six independent dimensions: Assertive, Cautious, Optimistic, Specific, Clear, and Relevant.
- Data Collection & Cleaning: They extracted 2,747 QA pairs from 120 Earnings Call Transcripts spanning from 2007 to 2021. A manual cleaning process removed verbal filler (e.g., "Hi, good morning") to focus strictly on substantive exchanges.
- Human Annotation: Each QA pair was rated on a 3-point scale (0: negative demonstration, 1: neutral/no relation, 2: positive demonstration) by three different annotators.
- Benchmarking: The authors evaluated both Pre-trained Language Models (PLMs) like BERT, RoBERTa, and FinBERT (via fine-tuning) and Large Language Models (LLMs) like Llama-3, Mixtral, and GPT-4o (via zero-shot prompting).
The resulting SubjECTive-QA dataset contains 49,446 annotations for 2,747 long-form QA pairs, covering companies listed on the NYSE across six major industries.
Key Insights and Results
The "Corporate Speak" Profile
The data reveals a distinct profile for corporate communication: company representatives are overwhelmingly "Clear" and "Relevant" (with 90% of answers rated a 2), but they frequently struggle with "Specificity." About 20% of answers were rated 0 for Specificity, confirming the hypothesis that executives often use vague language to protect company reputation or obscure unfavorable details.
The Independence of Subjectivity
A correlation analysis showed that the six chosen features are largely independent (Pearson correlations ranged from -0.08 to 0.39). This means a speaker can be "Optimistic" without necessarily being "Specific." Consequently, models can be fine-tuned to detect one specific trait without being heavily biased by the presence of another.
PLMs Still Punch Above Their Weight
Surprisingly, the best-performing model was RoBERTa-base, which achieved higher average weighted F1 scores (63.95%) than much larger models like Llama-3-70b. RoBERTa-base specifically led in detecting "Assertive" (49.10% F1) and "Optimistic" (62.69% F1) traits. This underscores the immense value of domain-specific fine-tuning over massive scale for nuanced classification tasks.
Furthermore, domain expertise matters. FinBERT-tone, a model pre-trained specifically on financial text, significantly outperformed general models on the "Specific" feature, which often requires an understanding of technical financial jargon.
Cross-Domain Utility
To test generalization, the authors used White House Press Briefings as an out-of-domain test set. Models trained on financial transcripts performed remarkably well on these political briefings (65.97% mean F1). This suggests that the linguistic markers of traits like "caution" or "assertiveness" remain consistent across different types of formal, high-stakes Q&A.
Limitations and Future Work
While SubjECTive-QA represents a significant step forward, the authors note a few limitations:
- Geographic and Medium Bias: The dataset is currently limited to US-listed companies and written transcripts. Original audio features—such as pitch, tone, and pauses—might provide even deeper insights into executive confidence that text alone misses.
- Annotator Subjectivity: Subjectivity is inherent in the task. The annotators were all male STEM students/researchers, which may introduce a specific demographic bias in how traits like "assertiveness" or "clarity" are perceived.
Looking ahead, the research team plans to expand the dataset to encompass other years and international markets. They also aim to explore the use of reinforcement learning to better capture the underlying intent behind non-transparent, "soft misinformation" answers.
Summary updated