Abstract
Consensus-based optimisation (CBO) is a versatile multi-particle metaheuristic optimisation method suitable for performing non-convex and non-smooth global optimisations in high dimensions. It has proven effective in various applications while at the same time being amenable to a theoretical convergence analysis. In this paper, we explore a variant of CBO, which incorporates truncated noise in order to enhance the well-behavedness of the statistics of the law of the dynamics. By introducing this additional truncation in the noise term of the CBO dynamics, we achieve that, in contrast to the original version, higher moments of the law of the particle system can be effectively bounded. As a result, our proposed variant exhibits enhanced convergence performance, allowing in particular for wider flexibility in choosing the noise parameter of the method as we confirm experimentally. By analysing the time evolution of the Wasserstein-2 distance between the empirical measure of the interacting particle system and the global minimiser of the objective function, we rigorously prove convergence in expectation of the proposed CBO variant requiring only minimal assumptions on the objective function and on the initialisation. Numerical evidences demonstrate the benefit of truncating the noise in CBO.
Introduction
Finding the global minimum of a non-convex, non-smooth function in high dimensions is a fundamental challenge in mathematics and machine learning. While metaheuristic methods like Particle Swarm Optimisation (PSO) are widely used, they often lack rigorous mathematical guarantees of convergence. Consensus-Based Optimisation (CBO) was developed to bridge this gap, utilizing a group of particles that move toward a "consensus point"—a weighted average of their positions where particles at lower function values carry more weight.
However, standard CBO faces a significant stability issue: if the random exploration noise is too high, particles can diverge toward infinity, creating heavy-tailed distributions. This makes the algorithm unpredictable and difficult to tune. This paper introduces a robust variant of CBO that "truncates" the noise term to prevent such instabilities.
Methodology
The proposed variant modifies the standard CBO dynamics by capping the noise term. In standard CBO, particles move based on a deterministic drift toward the consensus and a stochastic diffusion (noise) term that scales with the distance to that consensus.
The innovation lies in introducing a truncation parameter $M$. If the distance between a particle and the consensus point exceeds a certain threshold, the noise is truncated at a maximum value $M$. Additionally, a projection operator is introduced to ensure the consensus point remains within a reasonable search region. By capping the noise, the authors ensure the particle system's distribution remains sub-Gaussian rather than heavy-tailed, facilitating a more rigorous mathematical analysis using mean-field limits and Stochastic Differential Equations (SDEs).
Experimental Evaluation
The method was evaluated using standard benchmark optimization functions designed to trap algorithms in local minima, including the Ackley, Rastrigin, Griewank, Alpine, and Salomon functions. Tests were conducted in 15D and 20D spaces to simulate high-dimensional challenges.
Key Results and Robustness
A primary finding is the enhanced robustness of the truncated version. In standard CBO, the noise parameter ($\sigma$) must be carefully tuned; excessive noise causes divergence, while insufficient noise prevents exploration. Truncation acts as a safety net, allowing for larger noise parameters and better exploration without risking total divergence.
On the 15-dimensional Ackley function, the truncated version ($M=1$) achieved a near 100% success rate with only 300 particles. In contrast, the standard version ($M=\infty$) failed to reach a flawless success rate even with 1,200 particles. Phase diagrams, such as the one shown above, demonstrate that the truncated version has a significantly wider "sweet spot" for parameters, reducing the need for precise manual tuning.
Theoretical Insights
The "aha moment" of the research is the proof that truncation forces the particle law to be sub-Gaussian. This mathematical property allows the authors to prove convergence in expectation, a stronger result than many previous CBO studies. The paper reinforces the idea that CBO can be viewed as a derivative-free version of Stochastic Gradient Descent (SGD), capable of jumping over local energy barriers that would stall standard optimizers.
Conclusion and Future Work
While truncation significantly improves performance with isotropic noise, results for anisotropic noise were more nuanced. For certain functions like Rastrigin, the original version occasionally performed better with very low particle counts. Future research could explore adaptive truncation, where the parameter $M$ evolves as particles approach the global minimum. Despite these areas for exploration, the introduction of truncated noise provides a more stable and theoretically grounded framework for high-dimensional global optimization.
Implementation details and Matlab code are available at the authors' GitHub repository.
Summary updated