News Risk Alerting System (NRAS): A Data-Driven LLM Approach to Proactive Credit Risk Monitoring

Adil Nygaard, Ashish Upadhyay, Lauren Hinkle, Xenia Skotti, Joe Halliwell, Ian C Brown, Glen NoronhaEMNLP 2024

TL;DR

This paper introduces the News Risk Alerting System (NRAS), a production-ready system that uses LLMs to proactively identify credit-impacting news events for financial institutions. NRAS significantly automates credit risk monitoring by analyzing vast amounts of news data, alerting credit officers to potential risks with an estimated 77% precision.

What Problem Does This Paper Address?

Financial institutions need to constantly monitor the creditworthiness of their borrowers to mitigate potential losses. Traditionally, this involves periodic manual reviews of financial statements and news related to client companies. However, financial reports are infrequent, and manual news analysis is labor-intensive and can lead to delays in identifying critical events. This gap leaves institutions vulnerable to unexpected financial downturns in their clients. The paper addresses the challenge of efficiently and effectively identifying credit-relevant news in a timely manner from a massive, ever-growing stream of information.

Methods

NRAS employs a two-pronged approach:

  • LLM-based Text Analysis:
    • Targeted Sentiment Analysis: A fine-tuned BERT model analyzes news articles to assign a sentiment score (-1 to +1) to each focal company mentioned, reflecting the potential positive or negative impact on its financial standing. This is a "targeted" approach, meaning it focuses on the specific impact on each company, rather than a general sentiment of the article.
  • Mergers & Acquisitions (M&A) Classification: A DeBERTa-based binary classifier predicts the probability that a news article pertains to an M&A event. This is crucial as M&A activities can significantly alter a company's financial landscape.
  • Volumetric Analysis and Alert Generation:
    • Dynamic Thresholding: Instead of fixed alert triggers, NRAS uses a dynamic threshold based on a company's historical news volume over the past 90 days. This ensures that companies with lower typical news coverage can still trigger alerts, preventing bias towards larger, more frequently reported entities.
  • Alert Debouncing: To avoid alert fatigue, NRAS implements a "debouncing" mechanism. If an alert is raised, the threshold for subsequent alerts within seven days is doubled, requiring a significant escalation in news volume to trigger another alert for the same ongoing event.
  • Headline Deduplication and Ranking: To present concise and informative alerts, headlines are de-duplicated using LSH clustering, and the top four ranked headlines (based on sentiment or M&A score, and penalized for low-quality sources) are presented to credit officers.

Datasets

  • Sentiment Analysis: A dataset of 5348 news headlines sampled between 2019 and 2021 was used for fine-tuning the sentiment model. This dataset included 5194 headlines for training/validation and 154 for testing.
  • M&A Classification: A dataset of 1606 news articles from October 2021 to December 2022 was used for training and evaluating the M&A classifier. This dataset was split into training (60%), validation (20%), and testing (20%) sets, with a temporal split to account for the time-series nature of the data.

The paper does not introduce new benchmark datasets but utilizes existing news sources for its production system.

Key Insights and Ideas

  • Dynamic Thresholding is Crucial: The paper highlights that a static alert threshold is insufficient for effective credit risk monitoring. A dynamic approach, tailored to each company's news volume history, is essential for equitable and accurate alerting.
  • Targeted Sentiment Outperforms Generic: Focusing sentiment analysis on the specific impact on each company within an article, rather than a general article sentiment, leads to more relevant and actionable alerts.
  • M&A is a Distinct Credit Risk Factor: Recognizing M&A as a separate category of credit-impacting events, beyond just negative sentiment, allows for a more comprehensive risk assessment.
  • Production Deployment is Key: The paper emphasizes the successful deployment of NRAS as a production system, demonstrating its scalability and real-world utility, which is often a challenge for academic research.
  • End-User Feedback Loop: The system's continuous improvement is driven by feedback from Credit Officers (COs), who categorize alerts, providing valuable insights for system refinement.

Key Results

  • NRAS has been in production for nearly three years, alerting Credit Officers to over 2700 credit-relevant events.
  • The system achieves an estimated precision of 77%.
  • The sentiment model achieved a Mean Squared Error (MSE) of 0.0312 on the test set, outperforming other regression models.
  • The M&A classifier achieved a Macro-F1 score of 92.1 on the test set, with DeBERTa being the best-performing model.
  • The system processes over 20,000 news articles daily.
  • Alerts are categorized into five types: New Information, Recently Considered, Financial Factor, Other Factor, and Irrelevant. Approximately 21.31% of sentiment alerts and 27% of M&A alerts were classified as "New Information" by COs.

Limitations and Future Work

  • Concept Drift: The authors acknowledge that a limitation of NRAS is the potential for "concept drift," where new types of events may become relevant over time, requiring model updates. This is identified through discussions with SMEs, as the current evaluation setup might not detect it.
  • Future Enhancements:
    • Continuous evaluation and retraining of models with expanded datasets.
  • Increasing the number of alerting streams.
  • Integrating components for detailed information extraction (e.g., identifying buyers/sellers and deal size in M&A).
  • Leveraging recent advancements in GenAI for improved summarization and fine-grained information presentation to COs.
  • Extending the system to monitor individuals or countries, not just corporate entities.

Summary updated

BibTeX
@inproceedings{nygaard2024news,
  author = {Adil Nygaard and Ashish Upadhyay and Lauren Hinkle and Xenia Skotti and Joe Halliwell and Ian C Brown and Glen Noronha},
  title = {{News Risk Alerting System (NRAS): A Data-Driven LLM Approach to Proactive Credit Risk Monitoring}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
}
Figure