Query Dependent Model Selection and Adaptive Weighting

25 concepts18 papers3 proposals

In machine learning and information retrieval, we frequently search for a single, globally optimal model. We train a neural network or tune a retrieval formula to achieve the highest average performance across a benchmark dataset. However, this quest for a "one-size-fits-all" solution introduces a fundamental flaw: a model optimized for the average case inevitably underperforms on specific, heterogeneous inputs.

A search query looking for a specific website's homepage requires entirely different mathematical logic than a broad, exploratory query on a complex topic. Similarly, a visual question answering system encountering a medical image requires different specialized knowledge than one analyzing a casual smartphone photo.

Instead of forcing a single model to handle every edge case, or relying on static ensembles that average out individual strengths, a more powerful paradigm has emerged: dynamic, input-dependent model selection and weighting. The core intuition is to treat the input sample or query as a diagnostic signal. By analyzing the unique characteristics of an incoming input on the fly, a system can dynamically route it to the most capable model or compute a custom-tailored blend of specialized components.

Hard Selection via Pre-Retrieval Analysis

The earliest foundations of this idea were developed in Information Retrieval (IR) under the umbrella of "selective retrieval." Rather than executing multiple expensive retrieval models and merging their results post-hoc, early systems sought to analyze a query's statistical properties before running the full search pipeline.

Ben He et al. demonstrated this in their TREC 2003 experiments. They showed that by clustering queries based on their statistical features prior to retrieval, a system can automatically select the optimal term-weighting model for each specific query. This pre-retrieval strategy bypasses the computational overhead of post-retrieval adjustments while significantly boosting accuracy on traditionally "difficult" queries that fail under uniform ranking models.

[Incoming Query] ───► [Statistical Feature Extractor] ───► [Selection Mechanism]
                                                                  │
                                        ┌─────────────────────────┴─────────────────────────┐
                                        ▼                                                   ▼
                          [Term-Weighting Model A]                            [Term-Weighting Model B]

This capacity to swap underlying algorithms on the fly allows systems to adapt to user intent. For instance, Vassilis Plachouras et al. developed automated mechanisms to predict query difficulty and type, allowing their framework to dynamically switch between content-density models for broad topics and hyperlink-analysis models for homepage searches (University of Glasgow at TREC 2004).

Subsequent work expanded the features used to drive these hard-switching decisions. Jesus A. Rodriguez Perez et al. utilized the variance and spread of inverse document frequency (IDF) scores to optimize retrieval over noisy microblog data (University of Glasgow (UoG_TwTeam) at TREC Microblog 2013), while Richard McCreadie et al. incorporated spatial and category-based contexts to minimize search engine risk (University of Glasgow at TREC 2014). As James Allan et al. established, evaluating multiple retrieval models on a per-query basis allows a system to capitalize on the specialized strengths of individual algorithms far better than any static rank fusion method (Learning to select rankers).

Soft Blending and Adaptive Ensemble Weighting

While hard selection works well when query categories are distinct, many inputs occupy a grey area between domains. In these cases, completely switching from one model to another is too rigid. A softer, more robust approach is to dynamically adjust the weights of an ensemble based on the input context.

Siddhant Arora et al. investigated this by applying axiomatic features to determine dynamic per-query weights for retrieval models (Investigating Retrieval Method Selection with Axiomatic Features). They found that the optimal balance between exact keyword matching and semantic relevance shifts dramatically from query to query, meaning static ensembles inherently dilute the power of their constituent rankers.

To discover these weights without explicit supervision, Steven C. H. Hoi et al. introduced Semi-Supervised Ensemble Ranking (SSER). Instead of relying on static training weights, SSER analyzes the internal geometric structure of the unlabeled documents retrieved for a specific query (Semi-Supervised Ensemble Ranking). By evaluating how base rankers agree on the geometry of the local search space, the model computes query-dependent weights that adapt to the specific domain of the search.

This adaptive weighting strategy has been successfully applied across several distinct retrieval challenges:

Modern Extensions: Gating and Hypernetworks in Deep Learning

The core intuition of query-dependent selection has found a powerful second life in modern deep learning and large language models (LLMs). Instead of selecting classical retrieval formulas, modern architectures use auxiliary networks, gating functions, or hypernetworks to dynamically compose specialized neural parameters at inference time.

In parameter-efficient fine-tuning, arithmetic merging of specialized Low-Rank Adaptations (LoRAs) often leads to interference and performance degradation. To solve this, Xun Wu et al. proposed Mixture of LoRA Experts (MoLE). MoLE treats each layer's LoRA as an independent expert and implements a learnable gating function (Mixture of LoRA Experts). This function assigns dynamic, input-dependent weights to each adapter, ensuring that the model prioritizes different specialized capabilities for different parts of a prompt.

[Input Prompt] ───► [Gating Function] ───► [Dynamic Weights: [0.7, 0.2, 0.1]]
                           │
                           ├───► (0.7) ───► [LoRA Expert 1: Coding]  ──────┐
                           ├───► (0.2) ───► [LoRA Expert 2: Creative] ────┼─► [Blended Output]
                           └───► (0.1) ───► [LoRA Expert 3: Medical]  ─────┘

Similarly, Huishan Ji et al. applied an auxiliary routing model to the task of One-to-Many Visual Question Answering (Towards One-to-Many Visual Question Answering). Their framework analyzes input queries to dynamically blend task-specific "ability" adapters, allowing a single vision-language model to maintain high performance across highly divergent domains.

This paradigm also handles complex multimodal integration and foundation model prompting:

Why It Works: The Geometric Intuition

The consistent success of dynamic model selection across both classical IR and modern deep learning stems from a shared geometric reality: real-world data does not lie on a single, uniform manifold. Instead, data is composed of distinct, localized sub-manifolds, each requiring a different mathematical projection to optimize performance.

A static model or a uniformly weighted ensemble attempts to find a global compromise across this complex landscape. In doing so, it flattens the unique topological features of individual domains, leading to mediocre performance on difficult or niche inputs.

Dynamic selection and adaptive weighting function as a localized coordination layer. By treating the input's statistical features or text embeddings as a coordinate map, the system determines exactly where the input sits on the broader data manifold. It then conditions its processing architecture—whether by swapping a term-weighting formula or scaling neural attention weights—to match the local geometry of that specific input space. Ultimately, this approach allows systems to achieve the robustness of a generalist while leveraging the precision of a specialist.

Key Foundational Work

University of Glasgow at TREC 2004: Experiments in Web, Robust, and Terabyte Tracks with Terrier (2004) — contributed 2 concept elements to this direction.
University of Glasgow at the Robust Track- A Query-based Model Selection Approach for the Poorly-Performing Queries (2003) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_dynamic_model_selection_dcdde63b,
  title = {dynamic model selection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dynamic-model-selection/rdr_dcdde63bebfa4e8891f4cc793af53e11},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_dynamic_model_selection_dcdde63b_v1,
  title = {dynamic model selection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dynamic-model-selection/rdr_dcdde63bebfa4e8891f4cc793af53e11/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake