Iterative Denoising using Jensen-Renyi Divergences with an Application to Unsupervised Document Categorization

Damianos G. Karakos, Sanjeev Khudanpur, Jason Eisner, Carey E. PriebeICASSP 2007 · OpenReview

Abstract

Iterative denoising trees were used by Karakos et al. (2005) for unsupervised hierarchical clustering. The tree construction involves projecting the data onto low-dimensional spaces, as a means of smoothing their empirical distributions, as well as splitting each node based on an information-theoretic maximization objective. In this paper, we improve upon the work of (Karakos et al., 2005) in two ways: (i) the amount of computation spent searching for a good projection at each node now adapts to the intrinsic dimensionality of the data observed at that node; (ii) the objective at each node is to find a split which maximizes a generalized form of mutual information, the Jensen-Renyi divergence; this is followed by an iterative Naive Bayes classification. The single parameter α of the Jensen-Renyi divergence is chosen based on the "strapping" methodology, which learns a meta-classifier on a related task. Compared with the sequential information bottleneck method, our procedure produces state-of-the-art results on an unsupervised categorization task of documents from the "20 Newsgroups" dataset.
BibTeX
@inproceedings{karakos2007iterative,
  author = {Damianos G. Karakos and Sanjeev Khudanpur and Jason Eisner and Carey E. Priebe},
  title = {{Iterative Denoising using Jensen-Renyi Divergences with an Application to Unsupervised Document Categorization}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2007},
  url = {https://openreview.net/forum?id=OwJLgSWqHwZ},
}

Loading related content...

Figure