Cocktail: Learn a Better Neural Network Controller from Multiple Experts via Adaptive Mixing and Robust Distillation

Yixuan Wang, Chao Huang, Zhilu Wang, Shichao Xu, Zhaoran Wang, Qi Zhu

Introduction

Neural networks are increasingly utilized in the control of autonomous vehicles, smart buildings, and robotics. However, deploying these controllers in safety-critical cyber-physical systems (CPS) presents significant challenges regarding safety, robustness to noise, and formal verifiability. This paper introduces COCKTAIL, a framework designed to synthesize high-performance neural controllers by leveraging existing control experts—ranging from classical model-based controllers to pre-trained neural networks.

Methodology: Adaptive Mixing and Robust Distillation

The COCKTAIL framework operates in two distinct stages to produce a single, robust student network that is easier to verify and more efficient than its constituent experts.

The COCKTAIL framework: Adaptive mixing of control experts followed by robust distillation into a student network.

1. Adaptive Mixing via Reinforcement Learning

Rather than employing a discrete switching logic between experts, COCKTAIL utilizes a continuous mixing strategy. The weight assignment for each expert is formulated as a Markov Decision Process (MDP). A high-level policy, trained using Proximal Policy Optimization (PPO), observes the system state $s(t)$ and dynamically assigns weights to each expert. This "Teacher" controller is a weighted combination of all available experts, optimized for a balance of safety and energy efficiency.

2. Robust Distillation

To overcome the implementation and verification complexities of a mixture-of-experts model, COCKTAIL distills the Teacher into a single Student neural network ($\kappa^*$). Two key techniques are employed during this process:

  • Adversarial Training: The student is trained on adversarial examples generated via the Fast Gradient Sign Method (FGSM) to enhance resilience against perturbations.
  • Lipschitz Regularization: A penalty is added to the loss function to minimize the network's Lipschitz constant. A smaller Lipschitz constant ensures that the network's output remains stable under small input variations, which significantly accelerates formal verification tools.

Experimental Evaluation

The authors evaluated COCKTAIL on three non-linear control benchmarks: the Van der Pol Oscillator (2D), a 3D System, and the Cartpole System. The expert pool included both DDPG-trained neural networks and traditional model-based controllers.

Key Results and Insights

The distilled student network ($\kappa^*$) demonstrated superior performance compared to individual experts and state-of-the-art switching methods.

State-space trajectories and safe regions for the Van der Pol oscillator, comparing the COCKTAIL student ($\kappa^*$) against a standard distilled baseline ($\kappa_D$).

Robustness and Efficiency

In the Van der Pol Oscillator benchmark, COCKTAIL achieved a safe control rate of 98.8% under adversarial attack, significantly higher than the 85% achieved by the best single expert. Furthermore, energy efficiency improved dramatically; in the Cartpole system, average control energy was reduced from 106.1 (single expert) to 27.7.

Acceleration of Formal Verification

The most significant contribution is the reduction in verification time. For the oscillator system, formal verification of the COCKTAIL student took only 32 minutes, compared to 11 hours for a student distilled without Lipschitz regularization. In the 3D system, the un-regularized baseline caused memory faults during verification, while the COCKTAIL version was verified in minutes. This demonstrates that explicit optimization for "smoothness" makes formal safety proofs computationally tractable.

Conclusion

COCKTAIL provides a robust bridge between complex expert ensembles and verifiable single-network controllers. By combining adaptive reinforcement learning with regularized distillation, it produces controllers that are not only more efficient and robust but also orders of magnitude faster to formally verify, addressing a critical bottleneck in the deployment of neural networks for safety-critical systems.

Summary updated

BibTeX
@inproceedings{wang2021cocktail,
  author = {Yixuan Wang and Chao Huang and Zhilu Wang and Shichao Xu and Zhaoran Wang and Qi Zhu},
  title = {{Cocktail: Learn a Better Neural Network Controller from Multiple Experts via Adaptive Mixing and Robust Distillation}},
  booktitle = {DAC},
  year = {2021},
  doi = {10.1109/dac18074.2021.9586148},
}
Figure