Differentiable Programming à la Moreau

Vincent Roulet, Zaid Harchaoui

Abstract

The notion of a Moreau envelope is central to the analysis of first-order optimization algorithms for machine learning. Yet, it has not been developed and extended to be applied to a deep network and, more broadly, to a machine learning system with a differentiable programming implementation. We define a compositional calculus adapted to Moreau envelopes and show how to integrate it within differentiable programming. The proposed framework casts in a mathematical optimization framework several variants of gradient back-propagation related to the idea of the propagation of virtual targets.
BibTeX
@inproceedings{roulet2020differentiable,
  author = {Vincent Roulet and Zaid Harchaoui},
  title = {{Differentiable Programming à la Moreau}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  doi = {10.1109/icassp43922.2022.9746423},
}
Figure