Abstract: In this chapter, we focus on single-agent/model design. Section 3.1 begins with a discussion comparing end-to-end learning and feature engineering in the modern era. This section primarily explores the value of manual annotation and how to incorporate human insights into models to enhance the performance of downstream tasks. Beyond learning from humans and experts, the ability to learn from historical events and documents is also crucial for agents.
Introduction
While Large Language Models (LLMs) have revolutionized Natural Language Processing, they are not a "silver bullet" for specialized domains like finance. Current AI hype often suggests that prompt-based LLMs can entirely replace traditional supervised learning and human-curated datasets. However, general-purpose LLMs struggle with nuanced tasks such as estimating the "impact duration" of an event or performing causal inference during earnings calls. This research addresses the gap between general LLM capabilities and the requirements of domain-specific agents, exploring how to effectively integrate human expertise into model design without reverting to the labor-intensive feature engineering of the past.
Methodology
The authors propose a multi-pronged approach to single-agent design to bridge the performance gap:
- Pre-finetuning: An intermediate training stage between general pre-training and task-specific fine-tuning used to "inject" human-derived concepts (like numeral awareness) into the model.
- Expert-Hint Prompting: Providing "expert hints" (e.g., instructing the model to specifically analyze numbers in a tweet) to guide the LLM's attention.
- Co-trained RAG: For generating questions from earnings transcripts, a retriever is co-trained with a generator to identify key paragraphs requiring deep inference.
- Causal Knowledge Graph (KG) Integration: Enhancing LLM prompts with structured causal data (e.g., "sales" influencing "revenue") derived from financial reports.
- Model Editing (MEMIT): Using the Mass-Editing Memory in a Transformer (MEMIT) technique to modify specific internal layers to update knowledge without expensive full-scale retraining.
The Performance Gap and Human Insights
A key finding is that even advanced models like GPT-4 (few-shot) significantly underperform compared to smaller, supervised Pre-trained Language Models (PLMs) on specialized tasks like impact duration estimation. Pre-finetuning acts as a modern form of feature engineering, teaching a model a "novel concept" before the final task, making it much more adaptable to downstream financial forecasting.
Key results from these interventions include:
- Pre-finetuning Gains: Adopting an "impact duration" pre-finetuning task improved accuracy by 6.57% on stock movement prediction.
- The Power of Hints: Adding expert-designed hints to GPT-3.5 prompts for sentiment analysis resulted in an 11.25 point improvement in micro-F1 score.
- Retriever Efficiency: Co-trained retrievers raised the BERTScore for question generation from 2.158 to 4.700.
Model Editing and Scale Inconsistency
Model editing is a critical but complex component of agent design. A fascinating insight is that the type of question influences how a model reacts to being "edited." Models updated via fine-tuning show more stability when answering "why" questions compared to "when" or "where" questions, which suffer more from side effects.
Furthermore, findings from editing small models (like GPT-2) do not directly translate to larger LLMs (like LLaMA-2). Larger models exhibit much more drastic and unpredictable performance drops after knowledge updates, as seen in the comparative analysis of accuracy scores across different question categories.
Conclusion and Future Work
Model Editing remains a double-edged sword; while it allows for efficient updates, the side effects (performance degradation on unrelated tasks) are significant and vary based on question phrasing and model size. Additionally, no single Knowledge Graph works universally across all LLMs. Future research is needed to create more robust, model-agnostic ways to inject external knowledge and to explore how AI agents can eventually suggest their own creative features for human validation.
Summary updated