Learning multisensory representations for auditory-visual transfer of sequence category knowledge: a probabilistic language of thought approach
Author(s)
Yildirim, Ilker; Jacobs, Robert A.
Download13423_2014_734_ReferencePDF.pdf (220.5Kb)
PUBLISHER_POLICY
Publisher Policy
Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use.
Terms of use
Metadata
Show full item recordAbstract
If a person is trained to recognize or categorize objects or events using one sensory modality, the person can often recognize or categorize those same (or similar) objects and events via a novel modality. This phenomenon is an instance of cross-modal transfer of knowledge. Here, we study the Multisensory Hypothesis which states that people extract the intrinsic, modality-independent properties of objects and events, and represent these properties in multisensory representations. These representations underlie cross-modal transfer of knowledge. We conducted an experiment evaluating whether people transfer sequence category knowledge across auditory and visual domains. Our experimental data clearly indicate that we do. We also developed a computational model accounting for our experimental results. Consistent with the probabilistic language of thought approach to cognitive modeling, our model formalizes multisensory representations as symbolic “computer programs” and uses Bayesian inference to learn these representations. Because the model demonstrates how the acquisition and use of amodal, multisensory representations can underlie cross-modal transfer of knowledge, and because the model accounts for subjects’ experimental performances, our work lends credence to the Multisensory Hypothesis. Overall, our work suggests that people automatically extract and represent objects’ and events’ intrinsic properties, and use these properties to process and understand the same (and similar) objects and events when they are perceived through novel sensory modalities.
Date issued
2014-10Department
Massachusetts Institute of Technology. Department of Brain and Cognitive SciencesJournal
Psychonomic Bulletin & Review
Publisher
Springer US
Citation
Yildirim, Ilker, and Robert A. Jacobs. “Learning Multisensory Representations for Auditory-Visual Transfer of Sequence Category Knowledge: a Probabilistic Language of Thought Approach.” Psychon Bull Rev 22, no. 3 (October 23, 2014): 673–686.
Version: Author's final manuscript
ISSN
1069-9384
1531-5320