Spoken ObjectNet: A Bias-Controlled Spoken Caption Dataset
Name
CBMM-Memo-128.pdf
Size
2.91 MB
Format
Adobe PDF
Checksum (MD5)
e5a5f1bf00a8788bf705c5c74b54614f
Author(s)
Palmer, Ian
Rouditchenko, Andrew
Barbu, Andrei
Katz, Boris
Glass, James
Date Issued
August 30, 2021
Publisher
Center for Brains, Minds and Machines (CBMM), The 22nd Annual Conference of the International Speech Communication Association (Interspeech)
Series/Report no.
CBMM Memo;128
Abstract
Visually-grounded spoken language datasets can enable models to learn cross-modal correspon- dences with very weak supervision. However, modern audio-visual datasets contain biases that un- dermine the real-world performance of models trained on that data. We introduce Spoken ObjectNet, which is designed to remove some of these biases and provide a way to better evaluate how effec- tively models will perform in real-world scenarios. This dataset expands upon ObjectNet, which is a biascontrolled image dataset that features similar image classes to those present in ImageNet. We detail our data collection pipeline, which features several methods to improve caption quality, including automated language model checks. Lastly, we show baseline results on image retrieval and audio re- trieval tasks. These results show that models trained on other datasets and then evaluated on Spoken ObjectNet tend to perform poorly due to biases in other datasets that the models have learned. We also show evidence that the performance decrease is due to the dataset controls, and not the transfer setting.
Persistent DSpace Link
