Density-free variational inference fits an approximate posterior that we can only sample from. A neural network turns random noise into parameter samples, and it is trained without ever evaluating the probability density of what it produces. Training these samplers is fragile. Practitioners tune learning rates, auxiliary networks and loss weights by hand, and a natural idea is to borrow the tools that multi-task learning built for fragile composite losses, such as gradient normalization and gradient surgery.
This note argues that the obvious borrowing is subtly wrong, and that a narrower version is both correct and useful. In variational inference the weights on the loss terms are usually not free parameters: they define which distribution the method converges to. Reweighting the data-fit term against the entropy term quietly swaps the posterior for a "tempered" one. There is one kind of decomposition where reweighting is harmless: a discrepancy split into many sub-losses that are each zero at the true posterior. The clearest case is the set of one-dimensional random projections, or "slices," used by Sliced Wasserstein Variational Inference. We propose to balance only across such target-preserving components. We also propose to do the balancing directly on the samples, so it costs one backward pass instead of one per component. The goal is a sampler whose result is less sensitive to hyperparameters, without changing what it converges to.
Core Claim
We make two claims, one negative and one positive. The negative claim is that applying multi-task balancers to the standard split of the variational objective into an expected log-joint term and an entropy term biases the answer. At an optimum of the evidence lower bound, the gradient of the log-joint term with respect to the sampler parameters $\phi$ exactly cancels the gradient of the entropy term. Maximal directional conflict is therefore the defining property of the solution, not a sign of trouble. For two anti-parallel gradients $g_1=-c\,g_2$ with any $c>0$, the projection rule of Gradient Surgery for Multi-Task Learning returns zero for both. PCGrad's stationary set therefore contains the optimum of every weighted objective $w_1\mathcal{L}_1+w_2\mathcal{L}_2$, which is every tempered posterior. GradNorm moves the weights explicitly and lands on one of them, with the temperature set by the ratio of gradient norms rather than by the model.
The positive claim is that a sliced discrepancy with an adaptive distribution over projection directions keeps the posterior as its unique minimizer, as long as that distribution never loses full support. The objective is
$$D_\rho(q_\phi,\pi)=\mathbb{E}_{u\sim\rho}\big[W_2^2(u_\#q_\phi,\;u_\#\pi)\big],\qquad \rho=(1-\lambda)\,\mathrm{Unif}(S^{d-1})+\lambda\,\rho_{\text{bank}}.$$
Here $q_\phi$ is the sampler, $\pi$ the target posterior, $u_\#$ the projection onto direction $u$, $W_2$ the closed-form one-dimensional Wasserstein distance, $\rho_{\text{bank}}$ an adaptively weighted bank of directions, and $\lambda<1$ a fixed floor on the uniform part. Because the uniform component covers every direction, the Cramér–Wold theorem (a distribution is determined by all its one-dimensional projections) implies that $D_\rho=0$ only when $q_\phi=\pi$. Inside this safe family, the multi-task idea of up-weighting lagging components can speed up convergence and reduce tuning sensitivity without introducing bias. The claim to test is that it actually does so on the tasks where Sliced Wasserstein Variational Inference is currently used.
How This Relates to Prior Work
The density-free methods in the file motivate the problem. Operator Variational Inference optimizes a sampler against learned Stein test functions in a minimax game. Hierarchical Implicit Models and Likelihood-Free Variational Inference and Adversarial α-divergence Minimization use discriminators to estimate density ratios. The file reports saddle-point instability for all three. The adversarial α-divergence paper uses annealing, and still reports that the final choice of α trades RMSE against test log-likelihood.
Other methods avoid adversaries at the cost of inner loops. Unbiased Implicit Variational Inference runs Hamiltonian Monte Carlo inside each step. Kernelized Path Gradient Descent uses kernel score estimates whose cost grows quadratically in the number of samples. SWVI is the one method whose objective already decomposes into many cheap, individually target-preserving parts, so it supplies our base method. Its unweighted average over random slices is exactly the design choice we alter.
The balancing literature supplies the adaptive rules and also constrains how we use them. GradNorm contributes the training-rate criterion: weight each component by its current loss relative to its initial loss. Scalable Bilevel Loss Balancing contributes a first-order scheme with constant overhead, which shows that balancing need not scale with the number of components. Conflict-Averse Gradient Descent warns that MGDA stalls at arbitrary Pareto points, so we anchor to the average objective rather than seek Pareto stationarity.
All of these methods were built for multi-task networks, where any Pareto point is an acceptable answer. None of them asks whether the weighting changes what the answer means, and that question is the one that matters for inference. VIPLL is the closest paper touching both areas. It tunes its β and δ weights by hand, and in its setting those weights legitimately define the model, so it does not contradict the negative claim.
Research Program
The method keeps a bank of $M$ persistent directions alongside fresh uniform slices at every step. Every $T$ steps, a fraction of the bank is refreshed with the currently worst-matched fresh directions. Each bank direction $u_m$ carries a weight proportional to $(r_m)^{\alpha}$, where $r_m$ is its current one-dimensional Wasserstein loss divided by its loss when it entered the bank. This is GradNorm's inverse training rate applied to slices. Weights are normalized to sum to one and clipped from above.
The key step is how the balancing enters training. The gradient of each slice loss with respect to each sample is available in closed form: after sorting, it is the direction $u_m$ times the residual between the matched sorted values. We therefore sum the weighted per-slice sample gradients in sample space and backpropagate through the generator once. The balancing then costs a sort per slice and no extra backward passes. This sidesteps the per-task backward cost of PCGrad and CAGrad noted in Gap 4 of the file. Target samples come from SWVI's short MCMC chains, unchanged.
The analysis track proves the tempered-fixed-point result for PCGrad and GradNorm on the energy/entropy split. It also bounds how the upper clip on bank weights and the uniform floor λ control the bias from finite projection sets. The ablation track separates three factors: the adaptive weighting, the persistence of the bank, and the sample-space trick.
Baselines
The naive reference point is SWVI exactly as published, with a uniform average over $L$ fresh slices each step. We give it the same generator, MCMC chain length and optimizer as our method, and tune its learning rate and $L$ on the same grid. The file reports no number for SWVI on its toy, logistic-regression or MNIST experiments, so we rerun it.
The strongest external competitor on the shared benchmarks is KPG-IS from Kernelized Path Gradient Descent. It reports a Banana negative log-likelihood of 2.0913, against 3.6227 for KSIVI and 8.1287 for amortized Stein. On the 100-dimensional conditional diffusion benchmark it reports a log-marginal likelihood of 74528, against 70371 for Stein and 47853 for PVI. We will run KPG-IS with its released settings and our tuning grid, under matched wall-clock time. Beating it on Banana means a lower NLL than our reproduced KPG-IS at equal time. On the diffusion benchmark it means a higher log-marginal likelihood, or the same value in fewer iterations.
The compute-matched control is uniform SWVI given the extra sorts our method spends, converted into more slices. A second control, and the most serious threat, is a max-sliced-style variant. It keeps the same bank but always puts all weight on the currently worst direction, with no floor and no training-rate normalization.
The ablation that isolates the key ingredient uses the same bank with uniform weights. It separates "persistent directions" from "balanced directions." A second ablation keeps the adaptive weights but applies them through per-slice parameter-space gradients, to confirm that the sample-space trick gives the same result more cheaply.
For the negative claim, the baselines are PCGrad and GradNorm applied to the energy/entropy split of a UIVI-style objective. They are compared against the unmodified objective using a long-run HMC reference posterior.
Why not just…?
Why not just use more uniform slices? More slices reduce variance but do not change which directions dominate the gradient. In higher dimensions, most random directions see projections where the two distributions already look alike. The compute-matched uniform baseline tests this directly. If it matches our method at equal time, the proposal's positive claim fails, and we will report that outcome.
Why not just pick the worst-matched direction adversarially? This is a real threat. It is the natural reading of "prioritize lagging slices," and it gives up the target-preservation guarantee only when the uniform floor is dropped. The max-direction control is designed to settle the question. Our prediction is that pure max-direction training oscillates between directions and is more sensitive to the learning rate, because it reintroduces a minimax game. We also predict that the floored, rate-normalized version is more stable across the hyperparameter grid, even if the best-case accuracy of the two is similar. If the sensitivity curves coincide, we will present the method as a stabilized max-sliced variant rather than as new balancing.
Why not just apply PCGrad or GradNorm to the ELBO's two terms, since that is where the instability is? The fixed-point argument above says this converges to a tempered posterior. The experiment measures the mismatch in posterior variance against HMC on Bayesian logistic regression. If the balanced runs have systematically narrower or wider credible intervals than the unmodified objective at convergence, the negative claim is confirmed.
Why not just anneal the weights? Annealing only helps on the way to the final weighting. The adversarial α-divergence paper still reports a hard RMSE versus log-likelihood trade-off once the full weight is reached. Our balancing lives inside a discrepancy whose minimizer does not depend on the weights, so it has no final weighting to fix.
Proposed Experiments and Decisive Checks
Posterior accuracy is measured against long-run HMC reference samples, using held-out sliced Wasserstein distance and marginal credible-interval coverage. We also use the native metrics of each benchmark: NLL on Banana and the multimodal and X-shaped toys, test accuracy and log-likelihood for Bayesian logistic regression on Heart, Wine, Ionosphere and Waveform, and log-marginal likelihood on the 100-dimensional conditional diffusion benchmark. For latent-space inference on MNIST we use SWVI's pretrained-VAE setup and report complete-data log-likelihood. The file gives no SWVI number there, so UIVI's reported -94.09 test log-likelihood serves only as context, since it comes from a different protocol.
Hyperparameter sensitivity is the headline outcome. For each method we sweep learning rate, slice count, MCMC chain length and generator width. We report the fraction of configurations that reach within a fixed tolerance of the HMC reference, and the spread of final accuracy across seeds.
Three results would be decisive. First, the balanced method must beat compute-matched uniform SWVI on the sensitivity fraction, with no loss in best-case accuracy. Second, it must beat the max-direction control on stability. Third, the PCGrad- and GradNorm-on-ELBO runs must show measurable posterior-variance bias relative to HMC, while our method shows none beyond Monte Carlo error. If the first result fails, we will publish the second and third alone as a characterization of when multi-task balancers are safe inside inference.
References
- Gradient Surgery for Multi-Task Learning
- GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks
- Sliced Wasserstein Variational Inference
- Operator Variational Inference
- Hierarchical Implicit Models and Likelihood-Free Variational Inference
- Adversarial α-divergence minimization for Bayesian approximate inference
- Unbiased Implicit Variational Inference
- Semi-Implicit Variational Inference via Kernelized Path Gradient Descent
- LDC-MTL: Balancing Multi-Task Learning through Scalable Loss Discrepancy Control
- Conflict-Averse Gradient Descent for Multi-task learning
- Multi-Task Learning as Multi-Objective Optimization
- Amortized Variational Inference for Partial-Label Learning: A Probabilistic Approach to Label Disambiguation