Introduction
Few-shot learning is notoriously unstable, yet researchers often reach contradictory conclusions regarding the primary drivers of this volatility. This paper addresses the inconsistency by introducing a formal framework to decouple "randomness factors"—such as sample order and choice—revealing that these elements do not act in isolation. Instead, they interact significantly with each other and with systematic choices like prompt formatting and class count.
Methodology: Quantifying Factor Importance
The authors propose a rigorous investigation method to measure the relative importance of specific randomness factors while accounting for their dependencies. The process involves:
- Establishing a Golden Model: A baseline of total performance deviation is created by varying all randomness factors simultaneously.
- Mitigation Runs: To isolate a single factor (e.g., Sample Order), experiments are repeated while fixing non-investigated factors to a specific state.
- Importance Scoring: By comparing the "contributed" deviation of a factor against the "mitigated" background noise, an Importance Score is derived. A score above 0 indicates a significant driver of instability.
Key Insights and Interactions
A core finding is that interactions are the "missing link" in understanding few-shot sensitivity. For instance, the perceived impact of "Sample Order" in In-Context Learning (ICL) is frequently a side effect of the specific samples chosen. When "Sample Choice" is properly mitigated, the importance of order often diminishes.
The Role of Systematic Choices
Systematic decisions, such as prompt formatting, act as stability toggles. Minimalist prompts (e.g., input and label only) increase sensitivity to randomness, whereas "Optimized" or descriptive prompts provide a stabilizing effect, reducing the variance contributed by sample choice and order.
Furthermore, the study identifies a "More Classes, More Chaos" trend: as task complexity (number of classes) increases, Sample Order becomes a more dominant factor in ICL. Conversely, in fine-tuning scenarios, higher class counts tend to reduce the relative importance of initialization and data order.
Comparative Results across Architectures
The study evaluated a wide range of models, including Flan-T5, LLaMA-2-13B, Mistral-7B, and Zephyr-7B for ICL, alongside BERT and RoBERTa for fine-tuning.
Key findings include: * Sample Choice is Dominant: For ICL, Sample Choice was the most consistently influential factor, typically yielding an importance score around 0.25. * Fine-Tuning Stability: While generally more stable than ICL, fine-tuning (BERT/RoBERTa) remains sensitive to Data Split and Label Selection, with average importance scores of 0.52 and 0.34, respectively. * The Order Paradox: In binary tasks, Sample Order often showed negligible or even negative importance scores. However, in multi-class tasks (e.g., SNIPS with 7 classes), its importance surged to as high as 0.29.
Conclusion and Future Directions
While larger models like LLaMA-2-13B exhibit greater robustness to prompt changes, they remain susceptible to sample selection biases. The authors acknowledge that the computational cost of this framework is high, requiring hundreds of runs for accurate variance estimation. Future work may focus on automated sample selection algorithms and extending this analysis to more complex NLP tasks like summarization and question answering.
Summary updated