Rankmax: An adaptive projection alternative to the softmax function
Lists
Only you see this
Abstract
Several machine learning models involve mapping a score vector to a probability vector. Usually, this is done by projecting the score vector onto a probability simplex, and such projections are often characterized as Lipschitz continuous approximations of the argmax function, whose Lipschitz constant is controlled by a parameter that is similar to a softmax temperature. The aforementioned parameter has been observed to affect the quality of these models and is typically either treated as a constant or decayed over time. In this work, we propose a method that adapts this parameter to individual training examples. The resulting method exhibits desirable properties, such as sparsity of its support and numerically efficient implementation, and we find that it significantly outperforms competing non-adaptive projection methods. In our analysis, we also derive the general solution of (Bregman) projections onto the (n, k)-simplex, a result which may be of independent interest.
BibTeX
@inproceedings{kong2020rankmax,
author = {Weiwei Kong and Walid Krichene and Nicolas Mayoraz and Steffen Rendle and Li Zhang},
title = {{Rankmax: An adaptive projection alternative to the softmax function}},
booktitle = {Advances in Neural Information Processing Systems},
year = {2020},
url = {https://openreview.net/forum?id=-bP4PFwTVJ},
}Loading related content...