Analyzing Approximate Value Iteration Algorithms
Abstract
In this paper, we consider the stochastic iterative counterpart of the value iteration scheme wherein only noisy and possibly biased approximations of the Bellman operator are available. We call the aforementioned counterpart as the approximate value iteration (AVI) algorithm. The structure of AVI accounts for implementations with biased function approximations of the Bellman operator and sampling errors. This is pertinent since value iteration is combined with neural networks, which are used to approximate the Bellman operator, to solve complex problems that are susceptible to Bellman's curse of dimensionality. Further, instead of taking an expectation to calculate the Bellman operator, one generally uses samples. We present verifiable sufficient conditions under which AVI is stable (almost surely bounded) and converges to a fixed point of the approximate Bellman operator. We show that AVI can also be used in more general circumstances, i.e., for finding fixed points of contractive setvalued maps. To ensure the stability of AVI, we present three different yet related set of sufficient conditions that are based on the existence of an appropriate Lyapunov function. These Lyapunov function based conditions are easily verifiable and new to the literature. The verifiability is enhanced by the fact that a recipe for the construction of the necessary Lyapunov function is also provided. Finally, we show that the stability analysis of AVI can be readily extended to the general case of setvalued stochastic approximations.
 Publication:

arXiv eprints
 Pub Date:
 September 2017
 arXiv:
 arXiv:1709.04673
 Bibcode:
 2017arXiv170904673R
 Keywords:

 Electrical Engineering and Systems Science  Systems and Control;
 Mathematics  Dynamical Systems;
 Statistics  Machine Learning;
 62L20;
 93E35;
 37B25;
 34A60;
 90C39;
 37C25