Align or attend? Toward More Efficient and Accurate Spoken Word Discovery Using Speech-to-Image Retrieval
Lists
Only you see this
Abstract
Multimodal word discovery (MWD) is often treated as a byproduct of the speech-to-image retrieval problem. However, our theoretical analysis shows that some kind of alignment/attention mechanism is crucial for a MWD system to learn meaningful word-level representation. We verify our theory by conducting retrieval and word discovery experiments on MSCOCO and Flickr8k, and empirically demonstrate that both neural MT with self-attention and statistical MT achieve word discovery scores that are superior to those of a state-of-the-art neural retrieval system, outperforming it by 2% and 5% alignment F1 scores respectively.
BibTeX
@inproceedings{wang2021align,
author = {Liming Wang and Xinsheng Wang and Mark Hasegawa-Johnson and Odette Scharenborg and Najim Dehak},
title = {{Align or attend? Toward More Efficient and Accurate Spoken Word Discovery Using Speech-to-Image Retrieval}},
booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
year = {2021},
doi = {10.1109/icassp39728.2021.9414418},
}Loading related content...