This book concerns continuous-time controlled Markov chains, also known as continuous-time Markov decision processes. They form a class of stochastic control problems in which a single decision-maker wishes to optimize a given objective function. This book is also concerned with Markov games, where two decision-makers (or players) try to optimize their own objective function. Both decision-making processes appear in a large number of applications in economics, operations research, engineering, and computer science, among other areas.An extensive, self-contained, up-to-date analysis of basic optimality criteria (such as discounted and average reward), and advanced optimality criteria (e.g., bias, overtaking, sensitive discount, and Blackwell optimality) is presented. A particular emphasis is made on the application of the results herein: algorithmic and computational issues are discussed, and applications to population models and epidemic processes are shown.This book is addressed to students and researchers in the fields of stochastic control and stochastic games. Moreover, it could be of interest also to undergraduate and beginning graduate students because the reader is not supposed to have a high mathematical background: a working knowledge of calculus, linear algebra, probability, and continuous-time Markov chains should suffice to understand the contents of the book.
This book concerns continuous-time controlled Markov chains and Markov games. The former, which are also known as continuous-time Markov decision processes, form a class of stochastic control problems in which a single decision-maker has a wish to optimize a given objective function. In contrast, there are two or more decision-makers (or players, or controllers) trying to optimize its own objective function in a Markov game. Both decision-making processes appear in a large number of applications in economics, operations research, engineering, and computer science among other areas.
The main features of the control and game models studied in the book are the continuous time variable, the denumerable state space, and that the control (or action) sets are Borel spaces. Moreover, the transition and reward rates of the dynamical system may be unbounded. The authors are interested in some aspects of controlled Markov chains and Markov games such as characterizing the optimal reward functions, and determining optimal policies for each of the optimality criteria studied here. The main focus is on advanced optimality criteria (such as, bias, variance, sensitive discount, and Blackwell optimality), though they also deal with the basic optimality criteria (discounted and average reward).
A particular emphasis is made on the application of the results presented in this book. One of the main concerns is to propose assumptions on the control and game models that are easily verifiable (and verified) in practice. Moreover, algorithmic and computational issues are also analyzed. In particular, the authors propose approximation results that allow precise numerical approximations of the solution to some problems of practical interest. Applications to population models and epidemic processes are also shown.
