This book introduces the reader to the evolving area of simulation-based optimization, also known as simulation optimization. The book should serve as an accessible introduction to this topic and requires a background only in elementary mathematics. It brings the reader up to date on cutting-edge advances in simulation-optimization methodologies, including dynamic controls, also called Reinforcement Learning (RL) or Approximate Dynamic Programming (ADP), and static optimization techniques, e.g.,Simultaneous Perturbation, Nested Partitions, Backtracking Adaptive Search, Response Surfaces, and Meta-Heuristics. Special features of this book include:

Stochastic Control Optimization:

An Accessible Introduction to Reinforcement Learning Techniques for Solving Markov Decision Processes (MDPs), with Step-by-Step Descriptions of Numerous Algorithms, e.g., Q-Learning, SARSA, R-SMART, Actor-Critics, Q-P-Learning, and Classical Approximate Policy Iteration
A Detailed Discussion on Dynamic Programing for Solving MDPs and Semi-MDPs (SMDPs), Including Steps for Value Iteration and Policy Iteration
An Introduction to Function Approximation with Reinforcement Learning
An In-Depth Treatment of Reinforcement Learning Methods for SMDPs, Average Reward Problems, Finite Horizon Problems, and Two Time Scales
Computer Programs (available online)
A Gentle Introduction to Convergence Analysis via Banach Fixed Point Theory and Ordinary Differential Equations (ODEs)

Stochastic Static Optimization:

A Step-by-Step Description of Stochastic Adaptive Search Algorithms, e.g., Simultaneous Perturbation, Nested Partitions, Backtracking Adaptive Search, Stochastic Ruler, and Meta-Heuristics, e.g., Simulated Annealing, Tabu Search, and Genetic Algorithms
A Clear and Simple Introduction to the Methodology of Neural Networks

The book ends with a chapter on case studies that explain how these methods can be applied in real-world settings; an online repository of computer programs that can be downloaded from a website is also available.

The book was written for students and researchers in the fields ofengineering (industrial, electrical, and computer), computer science,operations research, management science, and applied mathematics. Anattractive feature of this book is its accessibility to readers new to this topic.

From the Author:

The main motivation for writing this book was to provide an accessible account of methods based on Reinforcement Learning (closely related to what is now also called Approximate Dynamic Programming) and Meta-Heuristics (closely related to what is now also called Stochastic Adaptive Search) for optimization in discrete-event systems via simulation. Reinforcement Learning (RL) is typically used for solving Markov decision problems (MDPs), which are dynamic optimization problems where the underlying discrete-event stochastic system is driven by Markov chains, while Meta-Heuristics are used for solving static optimization problems where the underlying system is any discrete-event stochastic system (not necessarily driven by Markov chains).
This book provides a selected collection of topics, mostly focused on model-free techniques, which are useful when one does not have access to the structure of the objective function (in static optimization) or the transition probability function (in dynamic optimization). My goal was neither to overwhelm the reader with mathematical details nor was it to cover every topic. Rather, the goal was to provide the reader with an overview of the fundamental concepts and at the same time provide the details required for solving real-world stochastic optimization problems via simulation-based techniques.
Some of the main topics covered are:

Reinforcement learning techniques for discounted and average reward MDPs
Detailed recipes for Reinforcement Learning algorithms such as Q-Learning, SARSA, R-SMART, and Actor Critics
Static optimization techniques rooted in meta-heuristics (simulated annealing, genetic algorithms, and tabu search) and stochastic adaptive search (nested partitions, stochastic ruler, and backtracking adaptive search) for discrete solution spaces and simultaneous perturbation for continuous solution spaces
Neural network algorithms useful for function approximation in response surface methods for static optimization and in reinforcement learning for MDPs with large state-action spaces
A detailed background on dynamic programming (value and policy iteration)
A special coverage of semi-MDPs (SMDPs), average reward problems, finite horizon MDPs, and two time scales in RL
A gentle introduction to convergence analysis of simulation optimization methods via Banach fixed point theory and Ordinary Differential Equations
Preview