(Smooth) Fictitious-play in identical-interest stochastic games with independent continuation-payoff estimates

Abstract

In this paper, we study fictitious-play-type dynamics for identical-interest stochastic games (SGs) and show their convergence to the Nash equilibrium. We develop off-policy and on-policy dynamics, and generalize these learning dynamics and convergence results to the smooth fictitious play variant when the smooth best-response is used in the updates. One key feature of our dynamics is the independent estimates of the continuation payoffs among agents. While this feature makes the dynamics more natural and uncoupled, it also leads to the challenge that the auxiliary stage games encountered during learning can become non-identical-interest anymore. We handle such a deviation from the identical-interest setting by either focusing on specific structures, e.g., the single-controller or symmetric SGs, or studying specific sublinear stepsizes to characterize the convergence rate of such a deviation as timestep evolves.