Representation learning of recipes

Abstract

This work introduces methods for learning distributed, vector representations of cooking recipes. The individual components of a recipe -- the images, instructions, and ingredients -- are first treated individually. These representations are learned from a large, multi-modal dataset collected -- and publicly released -- as part of this work. Their representations are then embedded in a joint vector space using a novel neural network model. Experiments on cross-modal retrieval and vector space arithmetic demonstrate the utility and generalizability of both the per-component and joint embeddings.