BCL: Bayesian In-Context Learning Framework for Information Extraction

Haoliang Liu, Chengkun Cai, Xu Zhao, Han Zhu, Shizhou Huang, Xinglin Zhang, Tao Chen, Jenq-Neng Hwang, Huaping Zhang, Lei Li

TL;DR

BCL (Bayesian In-Context Learning) is a novel prompt optimization framework that treats information extraction labels as controllable discrete variables. By using particle filtering and Bayesian updates to iteratively refine fine-grained semantic subcategories, BCL consistently boosts large language model performance across diverse scales and tasks without requiring parameter updates.

What Problem Does This Paper Address?

Information Extraction (IE) tasks, such as Named Entity Recognition (NER) and Relation Extraction (RE), are increasingly tackled using In-Context Learning (ICL) with Large Language Models (LLMs). However, current state-of-the-art methods suffer from two major flaws:

  1. Task Transfer Inconsistency: Methods that reframe IE as code generation or dialogue (e.g., CodeIE, ChatIE) perform poorly and inconsistently on smaller, open-source models, often experiencing catastrophic performance drops.
  2. Lack of Generalizability: Guideline-based approaches (e.g., GuideNER) rely on simplistic, frequency-based heuristics to select rules and are narrowly tailored to specific tasks like NER, failing to generalize to relation extraction.

The core challenge is bridging the gap between an LLM's broad prior knowledge and a dataset's specific, often coarse-grained annotation schema in a systematic, model-agnostic way.

Comparison of previous approaches vs BCL

Methods

The authors introduce BCL (Bayesian In-Context Learning), which reformulates prompt optimization as a control system problem. Instead of trying to optimize the intractable billions of parameters inside an LLM, BCL treats the prompt's rules as externally controllable state variables.

The workflow operates via an adaptive particle filtering algorithm across four main phases:

  1. Semantic Decomposition (Initialization): Coarse labels (like "Person") are broken down into atomic, fine-grained subcategories ("athlete", "public figure") to act as discrete "particles." Initial particles are generated by an LLM and assigned prior weights based on their linguistic naturalness (perplexity).
  2. Evaluation (Observation): The system evaluates how well each particle guides the LLM on a validation batch, computing a task-specific confidence/likelihood score.
  3. Weight Update: Particle weights are updated using a Bayesian formulation, multiplying the prior belief by the observed task performance.
  4. Multi-level Resampling (Control): To balance exploitation and exploration, BCL applies a two-tier strategy. Tier 1 prunes the bottom 50% of low-performing particles to maintain efficiency. Tier 2 applies LLM-guided semantic mutations (refinement, generalization, contextualization) to the surviving high-performing particles to discover even better variants.
BCL particle-based rule optimization pipeline

Datasets

The framework was evaluated across six widely recognized IE benchmarks spanning multiple domains:

  • Named Entity Recognition (NER): CoNLL-2003 (news), ACE 2005 (news/conversational), and GENIA (biomedical).
  • Relation Extraction (RE): NYT (news), CoNLL04 (general domain), and SciERC (scientific papers).

Key Insights and Ideas

  • Context as a Controllable Variable: By discretizing coarse labels into fine-grained semantic units, prompt engineering shifts from an ad-hoc art to a mathematically rigorous optimization problem.
  • Bypassing the Black Box: BCL proves you do not need access to a model's internal weights or gradients to optimize its behavior systematically; a low-dimensional interface of rules combined with sequential Monte Carlo methods is highly effective.
  • Extreme Data Efficiency: BCL achieves performance saturation using only 3–5% of the training data. The particle filtering mechanism rapidly updates belief distributions after each sample, preventing overfitting while drastically minimizing annotation burdens.
  • Cross-Model Transferability: The optimized rules capture generalized reasoning patterns rather than model-specific quirks. Rules optimized on small, accessible models (like Llama-3.1-8B) can be directly transferred to stronger, closed-source models (like GPT-3.5-turbo) to yield immediate performance gains.

Key Results

BCL consistently outperformed all baselines (One-shot, ChatIE, CodeIE, GuideNER) across all datasets and model scales (Qwen2.5-3B/7B, Llama-3.1-8B, Pixtral-12B).

  • Mitigating Catastrophic Degradation: While task-transfer methods like ChatIE and CodeIE completely failed on smaller models or RE tasks (often hitting 0.00 F1), BCL maintained robust performance. For instance, on CoNLL03 using Qwen-2.5-7B, BCL achieved 72.83 F1 compared to ChatIE's 25.64 F1 and GuideNER's 65.10 F1.
  • Cost-Efficient Deployment: A small model like Qwen-2.5-3B equipped with BCL (65.12 F1) matched the performance of a much larger Llama-3.1-8B one-shot baseline (65.78 F1) on CoNLL03, achieving comparable results with 62% fewer parameters.
Performance comparison across methods and task types
Detailed quantitative results across benchmarks

Limitations and Future Work

  • Optimization Overhead: The optimization phase requires $O(K \times M)$ iterations (where $K$ is the number of particles and $M$ is the number of training samples). This involves CPU-intensive numerical computations and multiple LLM inference calls, making it less ideal for real-time, frequently updated applications.
  • Particle Proliferation: Achieving rapid convergence generates a large number of particles, which increases the inference burden during the search phase.

The authors note that future research will focus on improving particle utilization efficiency to reduce these computational costs in practical settings.

Summary updated

Abstract

Existing information extraction (IE) tasks increasingly adopt in-context learning (ICL) with large language models. However, current approaches either show inconsistent performance across model scales or lack systematic optimization and generalizability. Building on this, we propose BCL (Bayesian In-Context Learning Framework for Information Extraction), the first optimization framework that uses particle filtering with Bayesian updates to systematically refine label representations across IE tasks. Through four steps initialization, observation, weight update, and resampling, BCL generalizes to both sequence labeling and relation classification paradigms. Extensive experiments demonstrate substantial and consistent improvements over existing approaches.
BibTeX
@inproceedings{liu2026bcl,
  author = {Haoliang Liu and Chengkun Cai and Xu Zhao and Han Zhu and Shizhou Huang and Xinglin Zhang and Tao Chen and Jenq-Neng Hwang and Huaping Zhang and Lei Li},
  title = {{BCL: Bayesian In-Context Learning Framework for Information Extraction}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1560},
}
Figure