In sequential decision-making, algorithms must constantly choose between exploiting the best-known option or exploring uncertain alternatives to gather more information. Classic strategies like Thompson Sampling (TS) and Upper Confidence Bound (UCB) rely on rigid mathematical frameworks—such as exact Bayesian updates or strict statistical confidence intervals—to dictate this balance.
While these classic formulations offer elegant theoretical guarantees, they often underperform in real-world applications. A rigid algorithm cannot adapt to varying time horizons, shifting business metrics, or the specific structural demands of a complex search space.
To bridge this gap, a powerful paradigm has emerged: parameterized exploration scaling. Instead of treating exploration as a fixed consequence of data density, this approach introduces explicit, tunable scaling knobs directly into the core mathematical mechanisms of exploration. By inflating or deflating these terms, practitioners can smoothly reshape the algorithm's behavioral landscape.
Deforming the Posterior: Variance Scaling in Thompson Sampling
To understand why scaling works, let us look at Thompson Sampling. Standard TS operates by maintaining a posterior distribution over the expected reward of each action, sampling a value from each distribution, and choosing the action with the highest sample. The width (variance) of the posterior represents the agent's uncertainty.
When an algorithm updates its beliefs strictly through Bayes' rule, the posterior variance narrows at a predetermined mathematical rate. However, Chapelle et al. demonstrated that manually scaling this posterior variance before sampling provides a powerful lever for performance tuning.
$$\sigma^2_{\text{sampled}} = \lambda \cdot \sigma^2_{\text{posterior}}$$
If we introduce a scaling parameter $\lambda$, we can alter the geometry of the selection process:
- When $\lambda < 1$ (Variance Compression): The posterior distribution is artificially sharpened. The sampler is forced to draw values closer to the empirical mean, suppressing exploration. This is highly effective in short-term horizons where immediate exploitation yields higher returns, though it risks getting stuck in local optima.
- When $\lambda > 1$ (Variance Inflation): The distribution flattens. Even if an arm has a well-characterized, mediocre history, inflation stretches the tails of its distribution, preserving the chance that a random sample will be exceptionally high. This forces the agent to continue exploring.
This intuition becomes vital in complex environments. For instance, Mazumdar et al. showed that scaling the posterior variance is essential when using approximate sampling methods (like Langevin algorithms) in high-dimensional spaces, preventing the distribution from concentrating too quickly and collapsing into sub-optimal exploitation. Similarly, Ding and Rong formalized this scaling within a behavioral framework, demonstrating that adjusting the stochastic term generalizes Thompson Sampling into a broad spectrum of exploration profiles that can even mimic human decision-making biases.
Expanding the Horizon: Confidence Radius Inflation in UCB
A parallel mechanism exists within deterministic, frequentist algorithms like Upper Confidence Bound (UCB). Standard UCB selects actions by maximizing an optimistic estimate of performance:
$$\text{Action} = \arg\max_{i} \left( \hat{\mu}_i + \alpha \cdot \text{bonus}_i \right)$$
Here, $\hat{\mu}_i$ is the empirical mean reward of arm $i$, and $\text{bonus}_i$ represents the uncertainty radius. In classic theory, $\alpha$ is a fixed constant derived from concentration inequalities to guarantee that the true mean lies below the bound with high probability.
In practice, scaling $\alpha$ transforms the algorithm's objective. This is particularly evident when balancing cumulative regret minimization (maximizing total rewards during the experiment) with best arm identification (finding the single best option as quickly as possible).
As Degenne et al. and Zhong et al. demonstrated, these two goals are fundamentally in tension. To find the absolute best arm with high confidence, an algorithm must continue pulling near-optimal arms to distinguish minor statistical differences, even if doing so incurs short-term regret. By treating $\alpha$ as a tunable inflation parameter, the system can smoothly interpolate between these two objectives.
Inflating the confidence radius prevents the algorithm from prematurely discarding candidates that are statistically close to the leader. As highlighted by Zhang et al., integrating an inflated bonus function ensures sufficient exploration of near-optimal arms, accelerating the definitive identification of the best option without suffering from the counterproductive over-exploration of completely sub-optimal choices.
Alternative Scaling Mechanisms: Pseudo-Rewards and Randomized Knobs
The principle of parameterized scaling extends beyond direct variance or bound multiplication. Researchers have engineered alternative structural knobs to achieve similar behavioral control:
- History Perturbation: Instead of altering the mathematical bounds after calculation, Kveton et al. proposed Perturbed-History Exploration (PHE). This approach injects synthetic "pseudo-rewards" directly into the historical data of an arm. The ratio of synthetic observations to actual observations acts as a scaling factor; a higher ratio increases the variance of the empirical mean estimate, elegantly forcing exploration without requiring complex posterior distributions.
- Randomized Hyperparameters: Rather than forcing practitioners to manually select a static scaling factor, Berk et al. (ijcai 2020) and Berk et al. (Knowl. Based Syst. 2020) introduced a framework where the exploration parameter itself is dynamically sampled from a parameterized distribution, such as a Gamma distribution. By adjusting the scale parameter of the hyperparameter distribution, users can safely inject domain knowledge about the expected smoothness of a target function, biasing the system toward aggressive exploitation or robust exploration.
Why It Works: Reshaping the Optimization Landscape
Ultimately, parameterized scaling works because it decouples the epistemic uncertainty (what the data tells us about our lack of knowledge) from the behavioral policy (how we choose to act on that lack of knowledge).
In complex real-world systems, this decoupling is essential. For instance, in e-commerce search ranking, Goswami et al. utilized a tunable exploration parameter to scale behavioral features against content features, preventing popular items from creating a "rich-get-richer" feedback loop. In structured environments like cascading bandits, Zhong et al. leveraged a tunable parameter $\lambda$ to dynamically balance information gathering against immediate recommendation relevance.
Even in complex engineering tasks like optimizing lithium-ion battery fast-charging, the authors of the STS-PBO framework showed that scaling parameters can be mapped to rate-distortion thresholds. Lowering the scaling value allows the system to tolerate higher sub-optimality, enabling broad exploration of the design space, while raising it eliminates sub-optimality tolerance to replicate traditional optimum-seeking behavior. Similarly, in multi-modal sampling problems like training Bayesian GANs, Pourshahrokhi et al. used scaling parameters to control whether a sampler explores within a detected mode or attempts to jump to entirely new modes.
By transforming rigid theoretical constants into flexible, parameterized knobs, this research direction provides a unified mathematical language for tailoring exploration to the precise constraints, horizons, and objectives of practical applications.
Key Foundational Work
An Empirical Evaluation of Thompson Sampling (2011) — contributed 3 concept elements to this direction.
Towards Optimization of e-Commerce Search and Discovery (2018) — contributed 1 concept elements to this direction.