The neural architecture of theory-based reinforcement learning
Name
2022.06.14.496001v1.full.pdf
Description
Submitted version
Size
15.98 MB
Format
Adobe PDF
Checksum (MD5)
50b6db49523c2c5d81216723be897876
Author(s)
Tomov, Momchil S
Tsividis, Pedro A
Pouncy, Thomas
Tenenbaum, Joshua B
Gershman, Samuel J
Date Issued
March 2023
Journal
Neuron
Publisher
Elsevier BV
Citation
Tomov, Momchil S, Tsividis, Pedro A, Pouncy, Thomas, Tenenbaum, Joshua B and Gershman, Samuel J. 2023. "The neural architecture of theory-based reinforcement learning." Neuron.
Version
Original manuscript
Abstract
Humans learn internal models of the world that support planning and generalization in complex environments. Yet it remains unclear how such internal models are represented and learned in the brain. We approach this question using theory-based reinforcement learning, a strong form of model-based reinforcement learning in which the model is a kind of intuitive theory. We analyzed fMRI data from human participants learning to play Atari-style games. We found evidence of theory representations in prefrontal cortex and of theory updating in prefrontal cortex, occipital cortex, and fusiform gyrus. Theory updates coincided with transient strengthening of theory representations. Effective connectivity during theory updating suggests that information flows from prefrontal theory-coding regions to posterior theory-updating regions. Together, our results are consistent with a neural architecture in which top-down theory representations originating in prefrontal regions shape sensory predictions in visual areas, where factored theory prediction errors are computed and trigger bottom-up updates of the theory.
MIT Department
Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences
Terms of Use
Creative Commons Attribution-NonCommercial-NoDerivs License
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1016/j.neuron.2023.01.023
