Transformer Module Networks for Systematic Generalization in Visual Question Answering
Name
CBMM-Memo-121.pdf
Size
1.06 MB
Format
Adobe PDF
Checksum (MD5)
f8a58e082f4366739963791126d54def
Author(s)
Yamada, Moyuru
D'Amario, Vanessa
Takemoto, Kentaro
Boix, Xavier
Sasaki, Tomotake
Date Issued
February 3, 2022
Publisher
Center for Brains, Minds and Machines (CBMM)
Series/Report no.
CBMM Memo;121
Abstract
Transformer-based models achieve great performance on Visual Question Answering (VQA). How- ever, when we evaluate them on systematic generalization, i.e., handling novel combinations of known concepts, their performance degrades. Neural Module Networks (NMNs) are a promising approach for systematic generalization that consists on composing modules, i.e., neural networks that tackle a sub-task. Inspired by Transformers and NMNs, we propose Transformer Module Network (TMN), a novel Transformer-based model for VQA that dynamically composes modules into a question-specific Transformer network. TMNs achieve state-of-the-art systematic generalization performance in three VQA datasets, namely, CLEVR-CoGenT, CLOSURE and GQA-SGL, in some cases improving more than 30% over standard Transformers.
Persistent DSpace Link
