Systematic Evaluation of Long-Context LLMs on Financial Concepts

Lavanya Gupta, Saket Sharma, Yiyun ZhaoEMNLP 2024

TL;DR

This paper systematically evaluates the performance of state-of-the-art long-context Large Language Models (LLMs) on financial tasks. They found that even advanced models like GPT-4o and GPT-4 Turbo struggle with long contexts, exhibiting "brittleness" and performance degradation as context length and task complexity increase, often leading to degenerate outputs. The authors advocate for more rigorous evaluation metrics and highlight the sensitivity of these models to prompt formatting.

What Problem Does This Paper Address?

Large Language Models (LLMs) are increasingly being developed with longer context windows, promising the ability to process and understand vast amounts of information. This is particularly relevant for real-world applications like financial analysis, where understanding lengthy documents is crucial. However, the actual reliability and effectiveness of these "long-context LLMs" (LC LLMs) in utilizing their extended context windows remain under-investigated. Existing benchmarks often use synthetic tasks or metrics that don't fully capture real-world performance, leading to an overestimation of LC LLM capabilities. This paper aims to bridge this gap by providing a systematic evaluation of leading LC LLMs on realistic financial tasks, examining how performance is affected by context length, task complexity, and prompt design.

Methods

The authors developed a novel framework for evaluating LC LLMs on financial concepts. This framework involves creating a dataset of real-world financial news articles and defining four progressively challenging tasks: Company Recognition (C), Time (T), Company+Time (CT), and Company+Sentiment (CS). They meticulously controlled various aspects of the evaluation, including:

  • Context Length: Testing across a range of context lengths from 4K to 128K tokens.
  • Task Difficulty: Varying complexity by combining single and multiple financial concepts.
  • Information Placement: Injecting "needles" (key information) at random positions within the context.
  • Hard Negatives: Including challenging distractors to simulate real-world noise.
  • Zero-Needle Scenarios: Testing the models' ability to correctly identify when relevant information is absent.
  • Prompt Engineering: Investigating the impact of prompt placement and formatting.

They evaluated two leading LC LLMs, GPT-4o and GPT-4-Turbo, using F1-scores and bootstrap confidence intervals for robust analysis.

Datasets

The paper introduces a newly created real-world financial news dataset for evaluating LC LLMs. This dataset is constructed from financial news articles and is designed to support four distinct tasks of varying difficulty. The authors emphasize that their dataset avoids the artificial dissimilarities often found in synthetic benchmarks, making it a more realistic testbed. They do not release the proprietary dataset due to confidentiality concerns.

Key Insights and Ideas

  • Brittleness at Scale: Even state-of-the-art LC LLMs exhibit significant performance degradation and "brittleness" as context length increases, especially for more complex tasks.
  • Task Complexity Matters: Performance drops dramatically with increasing task complexity. For multi-concept tasks like Company+Sentiment, models can collapse to near-zero performance at longer contexts.
  • Prompt Sensitivity: LC LLMs are surprisingly sensitive to minor variations in prompt formatting and the placement of instructions within the context window. Prepending instructions generally yields better results than appending them.
  • Beyond Recall: Simple recall metrics can be misleading for long-context retrieval. The authors advocate for using F1-scores and confidence intervals for more reliable and conclusive findings.
  • Degenerate Outputs: At longer contexts and higher task complexity, models can produce nonsensical or repetitive outputs, indicating a failure in instruction following.

Key Results

  • Performance Degradation: For the simplest task (Company recognition), F1-scores drop from nearly perfect (0.99) at 4K tokens to 0.4 at 128K tokens. For more complex tasks (Company+Sentiment), performance collapses to near zero at context lengths greater than 32K tokens.
  • Prompt Placement Impact: The "Append" prompt configuration was significantly worse than "Prepend" or "Prepend+Append" configurations. The "OpenAI Best Practices" configuration showed an overall improvement.
  • Degenerate Outputs: GPT-4-Turbo showed a higher vulnerability to degenerate outputs, with 4% on task C and 42% on task CS at 128K context length.
  • Zero-Needle Performance: Models are better at rejecting absent needles on easier tasks than on difficult ones.

Limitations and Future Work

  • Model Scope: The study focused only on GPT-class models due to organizational constraints, with plans to evaluate other model families in the future.
  • Generation Capabilities: The evaluation primarily focused on retrieval tasks and did not fully assess the long-form generation capabilities of LC LLMs, which are harder to evaluate precisely.
  • Instruction Complexity: The study began exploring the complexity of real-world instructions but did not exhaustively cover all variations.
  • Degenerate Output Investigation: The authors leave further investigation into the phenomenon of degenerate outputs to future work.

Summary updated

BibTeX
@inproceedings{gupta2024systematic,
  author = {Lavanya Gupta and Saket Sharma and Yiyun Zhao},
  title = {{Systematic Evaluation of Long-Context LLMs on Financial Concepts}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
}
Figure