Fictitious Play in Zero-Sum Stochastic Games
Name
21m1426675.pdf
Description
Published version
Size
472.63 KB
Format
Adobe PDF
Checksum (MD5)
e1ae226c494e3ef19ec1c87299067aa3
Author(s) • •
Sayin, Muhammed O
Parise, Francesca
Ozdaglar, Asuman
Date Issued
August 2022
Journal
SIAM Journal on Control and Optimization
Publisher
Society for Industrial & Applied Mathematics (SIAM)
Version
Final published version
Abstract
We present a novel variant of fictitious play dynamics combining classical fictitiousplay with Q-learning for stochastic games and analyze its convergence properties in two-player zero-sum stochastic games. Our dynamics involves players forming beliefs on the opponent strategyand their own continuation payoff (Q-function), and playing a greedy best response by using theestimated continuation payoffs. Players update their beliefs from observations of opponent actions.A key property of the learning dynamics is that update of the beliefs on Q-functions occurs at aslower timescale than update of the beliefs on strategies. We show that in both the model-based andmodel-free cases (without knowledge of player payoff functions and state transition probabilities),the beliefs on strategies converge to a stationary mixed Nash equilibrium of the zero-sum stochasticgame.
Terms of Use
Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use.
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1137/21M1426675