Abstract
Scaling training data for Large Language Model (LLM) prompt optimization is not a linear path to better performance. This paper identifies the "Implicit Context Wall," a phenomenon where excessive training examples in a prompt lead to diminishing or negative returns. To overcome this, we introduce TSGD-M (Textual Stochastic Gradient Descent with Momentum). By employing a performance-weighted sampling strategy, TSGD-M carries over insights from previous iterations without bloating the context window, enabling effective scaling and improved stability across complex reasoning tasks.
The Problem: The Implicit Context Wall
In "Compound AI Systems," LLMs are frequently used to optimize other LLMs through Textual Gradient Descent (TGD). In this paradigm, "gradients" are natural language feedback (e.g., "make the response more concise"). While numerical gradients scale efficiently with data, TGD faces two critical bottlenecks:
- The Implicit Context Wall: LLMs struggle to synthesize patterns when overwhelmed by too many examples. Performance typically peaks around 50 samples and drops sharply thereafter as the model loses focus on relevant feedback.
- Efficiency Drains: Increasing the number of examples causes inference costs to rise quadratically due to sequence length, making full-batch optimization prohibitively expensive.
Methodology: TSGD-M
To address these limitations, we propose TSGD-M, which adapts the classical optimization concept of momentum for the textual domain. Instead of simple concatenation, which leads to context overflow, TSGD-M uses a clever sampling mechanism.
- Momentum Sampling: The system maintains a history of past prompts and uses Gumbel-Top-k sampling to select a high-performing subset. This allows the optimizer to "remember" successful strategies without exceeding the context window.
- Performance-Based Decay: Prompts are weighted based on their validation accuracy; those with higher scores are more likely to be sampled for the next iteration.
- Block-wise Generation: New prompts are generated in blocks to maximize KV-cache reuse, significantly reducing computational overhead.
- Stochastic Minibatch Validation: To maintain cost-efficiency, prompts are evaluated on random data subsets, with a running mean used to stabilize the optimization signal.
Key Insights and Results
Our evaluation across benchmarks like MATH, GSM8K, and HotPotQA revealed several critical findings:
- Superior Scalability: While vanilla TGD fails to improve beyond 110 samples, TSGD-M continues to scale effectively up to 300 samples and beyond.
- Performance Gains: TSGD-M consistently outperformed baselines, yielding a 2% improvement on MATH and a 4% gain on TREC over the TextGrad framework.
- Stability: The momentum mechanism significantly reduced variance. On the MATH task, test accuracy variance dropped by 0.37%, resulting in more reliable prompt improvements.
- Framework Integration: TSGD-M is framework-agnostic and can be integrated into existing libraries such as DSPy (COPRO) and AdalFlow to provide immediate performance boosts.
Limitations and Future Work
Despite its success, TSGD-M remains dependent on the quality of the feedback LLM (e.g., GPT-4o). Weak feedback models can still introduce noise into the textual gradient. Additionally, while stochastic minibatches reduce costs, they may introduce bias if the validation set is unrepresentative. Future research will explore hierarchical optimization for multi-part pipelines and more complex memory structures for historical prompt retention.
Summary updated