Learning Cross-Modal Embeddings for Cooking Recipes and Food Images
Name
im2recipe.pdf
Description
Accepted version
Size
6.85 MB
Format
Adobe PDF
Checksum (MD5)
6d36519d90dbe120006f4ea6072e4b5c
Author(s)
Salvador, Amaia
Hynes, Nicholas
Aytar, Yusuf
Marin, Javier
Ofli, Ferda
Weber, Ingmar
Torralba, Antonio
Date Issued
November 2017
Journal
2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Citation
Salvador, Amaia et al. "Learning Cross-Modal Embeddings for Cooking Recipes and Food Images." 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017, Honolulu, Hawaii, USA, Institute of Electrical and Electronics Engineers (IEEE), November 2017 © 2017 IEEE
Version
Author's final manuscript
Abstract
In this paper, we introduce Recipe1M, a new large-scale, structured corpus of over 1m cooking recipes and 800k food images. As the largest publicly available collection of recipe data, Recipe1M affords the ability to train high-capacity models on aligned, multi-modal data. Using these data, we train a neural network to find a joint embedding of recipes and images that yields impressive results on an image-recipe retrieval task. Additionally, we demonstrate that regularization via the addition of a high-level classification objective both improves retrieval performance to rival that of humans and enables semantic vector arithmetic. We postulate that these embeddings will provide a basis for further exploration of the Recipe1M dataset and food and cooking in general. Code, data and models are publicly available.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1109/cvpr.2017.327
