This paper investigates how to make Large Language Model (LLM) processing more affordable through a technique called semantic filtering.
TL;DR
The authors introduce "cascadeability" to measure the theoretical maximum cost savings of using cheap "proxy" models to skip expensive LLM calls. Their study reveals that current systems often fail to reach this potential due to poor threshold calibration, leaving significant savings on the table.
What Problem Does This Paper Address?
Semantic filtering involves applying natural language predicates (e.g., "Does this legal document contain a covenant not to sue?") to massive document collections. While Large LLMs (the "oracle") are highly accurate for this, running them on every document is prohibitively expensive and slow.
To solve this, researchers use proxy cascades: a cheaper, smaller model (the proxy) first scores the documents. If the proxy is very confident, the system accepts its answer; if not, it "defers" to the expensive oracle. However, it’s often unclear why a system performs poorly. Is the small proxy model just not smart enough, or is the system failing to set the right confidence thresholds? This paper aims to disentangle these two bottlenecks.
Methods
The authors propose a unified framework that breaks the process into three phases:
- Proxy Construction: Building the cheap surrogate model (e.g., a smaller LLM or a trained lightweight model).
- Threshold Calibration: Using a small sample of oracle-labeled data to set thresholds ($\tau^-, \tau^+$) that satisfy a target accuracy (e.g., 95%).
- Cascaded Inference: Deciding when to skip the oracle based on those thresholds.
The paper's core contribution is Cascadeability ($\kappa_\alpha$). This is a retrospective diagnostic metric that calculates the maximum possible oracle reduction rate (ORR) a proxy could have achieved if the thresholds were perfectly optimized. By comparing this to the Achieved ORR, they identify the Realization Gap—the missed opportunity due to imperfect calibration.
Datasets
The authors compiled a benchmark of 10 real-world workloads across diverse domains including biomedical (PubMed), legal (Court, Legal), and user reviews (Steam). They also generated synthetic variants to test sensitivity to query selectivity (how many documents are actually "True") and oracle confidence (how hard the documents are for the LLM itself).
Key Insights and Ideas
- The Overconfidence Trap: Both small LLMs and lightweight models tend to be overconfident. Their internal probability scores don't reliably map to actual accuracy, making simple thresholding dangerous without careful recalibration.
- Two Distinct Bottlenecks: A system can have low savings because the proxy is weak (low cascadeability) OR because the calibration strategy is too conservative (large realization gap).
- Small LLMs Rule: Using a smaller version of the oracle LLM (e.g., Qwen-9B as a proxy for Qwen-27B) generally provides the best ranking quality, consistently outperforming distilled lightweight models.
- Selectivity Masks Weakness: In highly imbalanced datasets, a system can achieve high accuracy just by guessing the majority class. This can hide a proxy's inability to actually understand the content.
Key Results
- The Realization Gap is Significant: Systems often leave significant savings on the table. In some cases, the realization gap exceeded 40 percentage points, meaning the proxy could have skipped 40% more oracle calls while still meeting the accuracy target.
- System Trade-offs:
- LOTUS achieves high oracle reduction but can be unreliable on challenging tasks.
- BARGAIN is highly reliable (it almost always meets the accuracy target) but is often too conservative, leading to lower cost savings.
- Hidden Costs: The overhead of constructing proxies and calibrating thresholds is not negligible. For some systems, these "setup" costs actually exceeded the savings gained from skipping the oracle.
Limitations and Future Work
The authors identify several frontiers for future research:
- Long Documents: Handling documents that exceed an LLM's context window requires complex chunking and aggregation.
- Reasoning-Intensive Tasks: Some predicates require "Chain-of-Thought" reasoning, which is expensive even for smaller proxies.
- Data Drift: In production, document collections evolve. Systems need ways to detect when their calibrated thresholds are no longer valid.
- Multimodal Filtering: Extending these cascades to images, audio, and video remains an open challenge.
Summary updated