The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision
Name
1904.12584.pdf
Description
Accepted version
Size
7.77 MB
Format
Adobe PDF
Checksum (MD5)
93644435599dab340080daaa1f7611e2
Author(s)
Mao, Jiayuan
Gan, Chuang
Kohli, Pushmeet
Tenenbaum, Joshua B
Wu, Jiajun
Date Issued
May 2019
Journal
ICLR 2019: International Conference on Learning Representations
Publisher
International Conference on Learning Representations
Citation
Mao, Jiayuan et al. "The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision." ICLR 2019: 7th International Conference on Learning Representations, May 6-9, 2019, New Orleans, Louisiana: https://openreview.net/forum?id=rJgMlhRctm ©2019 Author(s)
Version
Author's final manuscript
Abstract
We propose the Neuro-Symbolic Concept Learner (NS-CL), a model that learns visual concepts, words, and semantic parsing of sentences without explicit supervision on any of them; instead, our model learns by simply looking at images and reading paired questions and answers. Our model builds an object-based scene representation and translates sentences into executable, symbolic programs. To bridge the learning of two modules, we use a neuro-symbolic reasoning module that executes these programs on the latent scene representation. Analogical to human concept learning, the perception module learns visual concepts based on the language description of the object being referred to. Meanwhile, the learned visual concepts facilitate learning new words and parsing new sentences. We use curriculum learning to guide the searching over the large compositional space of images and language. Extensive experiments demonstrate the accuracy and efficiency of our model on learning visual concepts, word representations, and semantic parsing of sentences. Further, our method allows easy generalization to new object attributes, compositions, language concepts, scenes and questions, and even new program domains. It also empowers applications including visual question answering and bidirectional image-text retrieval.
MIT Department
Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory
MIT-IBM Watson AI Lab
Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike
Persistent DSpace Link
DOI of Published Version
https://openreview.net/forum?id=rJgMlhRctm
