Improved Regret Bounds of (Multinomial) Logistic Bandits via Regret-to-Confidence-Set Conversion

Junghyun Lee, Se-Young Yun, Kwang-Sung Jun

Abstract

Online continual learning faces the challenge of representation shift, where updating an encoder to learn new data causes the feature mappings of old data to drift, leading to catastrophic forgetting. We propose DeepCCG (Deep Class-Conditional Gaussians), a method that replaces standard softmax classifiers with a Bayesian class-conditional Gaussian model. This allows for instantaneous realignment of the classifier to shifting representations in a single step. Combined with a Log Conditional Marginal Likelihood loss and information-theoretic sample selection, DeepCCG significantly reduces forgetting and outperforms state-of-the-art methods on standard benchmarks like CIFAR and MiniImageNet.

Introduction: The Challenge of Representation Shift

In online continual learning, models typically consist of an encoder that creates a representation of the data and a classifier that assigns labels. A major hurdle is representation shift: as the encoder updates to learn new information, the "location" of old data in the feature space moves.

Standard classifiers (like a linear layer followed by a softmax) adapt to these shifts slowly via gradient descent. This creates a period of "misalignment" where the classifier is looking for old classes in the wrong place. This noise during learning often forces the encoder to "forget" old information unnecessarily to satisfy an outdated classifier.

Methods: Deep Class-Conditional Gaussians

DeepCCG introduces three key innovations to maintain alignment between the feature extractor and the classifier:

1. Bayesian Class-Conditional Gaussian Classifier

Instead of a standard neural network head, DeepCCG models each class as a Gaussian distribution in the representation space. Because this is a tractable Bayesian model, the classifier's decision boundaries (the posterior) can be recomputed in a single step whenever the representation shifts. This "instantaneous" adaptation prevents the encoder from receiving noisy signals.

DeepCCG enables fast adaptation to representation shift compared to standard CL methods.

2. Log Conditional Marginal Likelihood (LCML) Loss

To train the encoder, the authors use a loss function that maximizes the probability of labels given the representation, conditioned on stored memory. This ensures the representation is optimized specifically to be well-separated for the Gaussian classifier.

3. Information-Theoretic Sample Selection

To maintain a fixed-size memory buffer, DeepCCG selects examples that minimize the KL divergence between the "true" posterior (based on all seen data) and the posterior induced by the subset in memory. This ensures the stored samples are the most representative of the class means.

Experimental Results

The method was evaluated on three standard computer vision benchmarks: CIFAR-10, CIFAR-100, and MiniImageNet. DeepCCG consistently outperformed a wide spectrum of competitive baselines, including ER-reservoir, DER++, and iCaRL.

  • Accuracy Boost: On the CIFAR-100 Class-Incremental benchmark, DeepCCG achieved a mean accuracy of 18.00%, significantly higher than the next best method, ER-ACE (13.51%).
  • Stability: Analysis showed that for any given amount of shift in the feature space, DeepCCG’s accuracy dropped significantly less than other methods, proving it is more robust to the internal "moving targets" of neural network training.
  • Efficiency: Because the classifier update is a single-step calculation rather than an iterative process, it maintains high performance without adding significant computational overhead during the update phase.

Limitations and Future Work

While effective, DeepCCG assumes no interaction between the mean parameters of different classes, which could lead to clashes in dense class-incremental settings. Additionally, the model requires a "critical mass" of samples to accurately estimate Gaussian means; performance drops more sharply when the memory buffer is very small. Future work may explore learning covariance matrices alongside means to increase expressive power.

Conclusion

DeepCCG demonstrates that alignment is the antidote to forgetting. By making the classifier instantaneously adaptable, the system prevents the encoder from receiving noisy signals, leading to more stable and effective continual learning.

Summary updated

Abstract

Logistic bandit is a ubiquitous framework of modeling users' choices, e.g., click vs. no click for advertisement recommender system. We observe that the prior works overlook or neglect dependencies in S≥‖θ⋆‖2, where θ⋆∈ℝd is the unknown parameter vector, which is particularly problematic when S is large, e.g., S≥d. In this work, we improve the dependency on S via a novel approach called {\it regret-to-confidence set conversion (R2CS)}, which allows us to construct a convex confidence set based on only the \textit{existence} of an online learning algorithm with a regret guarantee. Using R2CS, we obtain a strict improvement in the regret bound w.r.t. S in logistic bandits while retaining computational feasibility and the dependence on other factors such as d and T. We apply our new confidence set to the regret analyses of logistic bandits with a new martingale concentration step that circumvents an additional factor of S. We then extend this analysis to multinomial logistic bandits and obtain similar improvements in the regret, showing the efficacy of R2CS. While we applied R2CS to the (multinomial) logistic model, R2CS is a generic approach for developing confidence sets that can be used for various models, which can be of independent interest.
BibTeX
@inproceedings{lee2024improved,
  author = {Junghyun Lee and Se-Young Yun and Kwang-Sung Jun},
  title = {{Improved Regret Bounds of (Multinomial) Logistic Bandits via Regret-to-Confidence-Set Conversion}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2024},
  url = {https://openreview.net/forum?id=vsOmJQxfR3},
}
Figure