Exploring Language Models to Analyze Market Demand Sentiments from News

Tirthankar Dasgupta, Manjira SinhaACL 2024

TL;DR

This paper introduces a novel TwinBERT-based framework for automatically extracting market demand signals from news articles. By jointly performing subject-of-demand entity extraction and sentiment analysis, and then augmenting this with causal analysis, the system provides richer insights for decision-makers than existing methods.

What Problem Does This Paper Address?

Accurate demand forecasting is crucial for effective supply chain planning, budgeting, and marketing. Traditionally, this involves manual analysis of news and analyst reports by experts, a process that is becoming increasingly difficult and time-consuming due to the sheer volume of available information. While social media offers short-term demand insights, long-term assessment still relies on business news. This paper aims to automate the extraction of demand-related information from these business texts, moving beyond simple sentiment analysis to understand what is driving demand changes.

Methods

The core of the proposed solution is a TwinBERT architecture that leverages pre-trained BERT models for two distinct but related tasks:

  • Subject-of-Demand (SDE) Extraction: A sequence labeling model (BERT₁) identifies specific product or service entities mentioned in relation to demand.
  • Sentiment Polarity Classification: A classification model (BERT₂) determines whether the sentiment associated with an SDE indicates a rise (positive) or fall (negative) in demand.

These two tasks are trained jointly. The system also incorporates a causal analytics module (based on a linguistically informed BiLSTM architecture) to identify the reasons behind demand fluctuations, extracting cause-and-effect relationships from the text. The outputs from the demand extraction/sentiment analysis and the causal analysis modules are then merged to provide a comprehensive view.

Datasets

The authors curated a dataset of approximately 74,150 news documents spanning ten years (2012-2021) from the automobile and housing sectors. From this, they extracted around 29,000 documents containing demand-related keywords. This dataset was then manually annotated by six annotators for SDEs and sentiment polarity. The final gold-standard dataset consists of 12,200 text documents, with 7,400 from the automobile domain and 4,800 from the housing domain. This dataset comprises roughly 43% positive, 37% negative, and 20% neutral demand samples. 70% of this data was used for training, and 30% for testing.

Key Insights and Ideas

  • Beyond Simple Sentiment: The paper emphasizes that understanding why demand changes (causality) is as important as knowing if it's rising or falling.
  • Joint Training Advantage: Combining SDE extraction and sentiment classification into a single, jointly trained framework (TwinBERT) leads to better performance than training them separately.
  • Contextual Nuance: Business texts can be complex, and accurately identifying the specific product (SDE) and its associated sentiment requires sophisticated NLP techniques.
  • Automating Expert Analysis: The goal is to empower decision-makers by automating the laborious process of sifting through news for demand signals, providing structured and actionable insights.

Key Results

The proposed TwinBERT model significantly outperforms existing baseline systems.

  • SDE Extraction: Achieved an F1 score of 0.81 (0.79 precision, 0.83 recall) for the automobile domain and 0.79 (0.77 precision, 0.81 recall) for the housing domain.
  • Sentiment Polarity Classification: Achieved an F1 score of 82% for the automobile dataset and 75.8% (73% precision, 79% recall) for the housing dataset.
  • LLM Comparison: When compared to fine-tuned Mistral-7B and few-shot LLAMA-2, the TwinBERT model demonstrated superior performance, particularly in recall and F1 scores, suggesting that its specialized architecture is more adept at this domain-specific task than general-purpose LLMs without extensive fine-tuning.

Limitations and Future Work

  • Implicit Demands: The current work does not identify implicit demands, meaning demands that are not explicitly stated but can be inferred.
  • LLM Performance Analysis: Further in-depth analysis of LLMs' performance on demand mining tasks is recommended.
  • Causal Model Evaluation: The paper acknowledges that the evaluation of the causal models themselves was not explicitly discussed in detail.

Future work could involve extending the framework to social media data and exploring more advanced LLM integration for improved performance.

Summary updated

BibTeX
@inproceedings{dasgupta2024exploring,
  author = {Tirthankar Dasgupta and Manjira Sinha},
  title = {{Exploring Language Models to Analyze Market Demand Sentiments from News}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure