TL;DR
BCL (Bayesian In-Context Learning) is a novel prompt optimization framework that treats information extraction labels as controllable discrete variables. By using particle filtering and Bayesian updates to iteratively refine fine-grained semantic subcategories, BCL consistently boosts large language model performance across diverse scales and tasks without requiring parameter updates.
What Problem Does This Paper Address?
Information Extraction (IE) tasks, such as Named Entity Recognition (NER) and Relation Extraction (RE), are increasingly tackled using In-Context Learning (ICL) with Large Language Models (LLMs). However, current state-of-the-art methods suffer from two major flaws:
- Task Transfer Inconsistency: Methods that reframe IE as code generation or dialogue (e.g., CodeIE, ChatIE) perform poorly and inconsistently on smaller, open-source models, often experiencing catastrophic performance drops.
- Lack of Generalizability: Guideline-based approaches (e.g., GuideNER) rely on simplistic, frequency-based heuristics to select rules and are narrowly tailored to specific tasks like NER, failing to generalize to relation extraction.
The core challenge is bridging the gap between an LLM's broad prior knowledge and a dataset's specific, often coarse-grained annotation schema in a systematic, model-agnostic way.
Methods
The authors introduce BCL (Bayesian In-Context Learning), which reformulates prompt optimization as a control system problem. Instead of trying to optimize the intractable billions of parameters inside an LLM, BCL treats the prompt's rules as externally controllable state variables.
The workflow operates via an adaptive particle filtering algorithm across four main phases:
- Semantic Decomposition (Initialization): Coarse labels (like "Person") are broken down into atomic, fine-grained subcategories ("athlete", "public figure") to act as discrete "particles." Initial particles are generated by an LLM and assigned prior weights based on their linguistic naturalness (perplexity).
- Evaluation (Observation): The system evaluates how well each particle guides the LLM on a validation batch, computing a task-specific confidence/likelihood score.
- Weight Update: Particle weights are updated using a Bayesian formulation, multiplying the prior belief by the observed task performance.
- Multi-level Resampling (Control): To balance exploitation and exploration, BCL applies a two-tier strategy. Tier 1 prunes the bottom 50% of low-performing particles to maintain efficiency. Tier 2 applies LLM-guided semantic mutations (refinement, generalization, contextualization) to the surviving high-performing particles to discover even better variants.
Datasets
The framework was evaluated across six widely recognized IE benchmarks spanning multiple domains:
- Named Entity Recognition (NER): CoNLL-2003 (news), ACE 2005 (news/conversational), and GENIA (biomedical).
- Relation Extraction (RE): NYT (news), CoNLL04 (general domain), and SciERC (scientific papers).
Key Insights and Ideas
- Context as a Controllable Variable: By discretizing coarse labels into fine-grained semantic units, prompt engineering shifts from an ad-hoc art to a mathematically rigorous optimization problem.
- Bypassing the Black Box: BCL proves you do not need access to a model's internal weights or gradients to optimize its behavior systematically; a low-dimensional interface of rules combined with sequential Monte Carlo methods is highly effective.
- Extreme Data Efficiency: BCL achieves performance saturation using only 3–5% of the training data. The particle filtering mechanism rapidly updates belief distributions after each sample, preventing overfitting while drastically minimizing annotation burdens.
- Cross-Model Transferability: The optimized rules capture generalized reasoning patterns rather than model-specific quirks. Rules optimized on small, accessible models (like Llama-3.1-8B) can be directly transferred to stronger, closed-source models (like GPT-3.5-turbo) to yield immediate performance gains.
Key Results
BCL consistently outperformed all baselines (One-shot, ChatIE, CodeIE, GuideNER) across all datasets and model scales (Qwen2.5-3B/7B, Llama-3.1-8B, Pixtral-12B).
- Mitigating Catastrophic Degradation: While task-transfer methods like ChatIE and CodeIE completely failed on smaller models or RE tasks (often hitting 0.00 F1), BCL maintained robust performance. For instance, on CoNLL03 using Qwen-2.5-7B, BCL achieved 72.83 F1 compared to ChatIE's 25.64 F1 and GuideNER's 65.10 F1.
- Cost-Efficient Deployment: A small model like Qwen-2.5-3B equipped with BCL (65.12 F1) matched the performance of a much larger Llama-3.1-8B one-shot baseline (65.78 F1) on CoNLL03, achieving comparable results with 62% fewer parameters.
Limitations and Future Work
- Optimization Overhead: The optimization phase requires $O(K \times M)$ iterations (where $K$ is the number of particles and $M$ is the number of training samples). This involves CPU-intensive numerical computations and multiple LLM inference calls, making it less ideal for real-time, frequently updated applications.
- Particle Proliferation: Achieving rapid convergence generates a large number of particles, which increases the inference burden during the search phase.
The authors note that future research will focus on improving particle utilization efficiency to reduce these computational costs in practical settings.
Summary updated