Simulation-Based Optimization: Parametric Optimization Techniques and Reinforcement Learning (Operations Research/Computer Science Interfaces Series) - Hardcover

Abhijit Gosavi

Hardcover

ISBN 10: 1402074549 ISBN 13: 9781402074547

Publisher: Springer, 2003

Synopsis

This book introduces the evolving area of simulation-based optimization. Since it became possible to analyze random systems using computers, scientists and engineers have sought the means to optimize systems using simulation models. Only recently, however, has this objective had success in practice. Cutting-edge work in computational operations research, including dynamic programming, e.g., Reinforcement Learning (RL) or Approximate Dynamic Programming (ADP), and static optimization via Stochastic Adaptive Search, e.g.,  Simultaneous Perturbation and Meta-Heuristics, has made it possible to use simulation in conjunction with optimization techniques.  Some special features of the book include:

• An Accessible Introduction to Reinforcement Learning Techniques for Solving Markov Decision Processes (MDPs)
• A Step-by-Step Description of Stochastic Adaptive Search Algorithms, e.g., Simultaneous Perturbation, Simulated Annealing, Tabu Search, and Genetic Algorithms, for Static Simulation-Based Optimization 
• A Clear and Simple Introduction to the Methodology of Neural Networks
• A Gentle Introduction to Convergence Analysis of a Subset of Methods Enumerated Above
• A Clear Discussion on Dynamic Programing for Solving MDPs and Semi-MDPs (SMDPs)
• An In-Depth Treatment of RL Methods for SMDPs and Average Reward Problems
• Computer Programs

This book is written for students and researchers in the fields of engineering (industrial, electrical, and computer), computer science, operations research, management science, and applied mathematics.  An attractive feature of this book is its accessibility to readers new to this topic.

About this title

From the Author

The main motivation for writing this book was to provide an accessible account of methods based on Reinforcement Learning (closely related to what is now also called Approximate Dynamic Programming) and Meta-Heuristics (closely related to what is now also called Stochastic Adaptive Search) for optimization in discrete-event systems via simulation.  Reinforcement Learning (RL) is typically used for solving Markov decision problems (MDPs), which are dynamic optimization problems where the underlying discrete-event stochastic system is driven by Markov chains, while Meta-Heuristics are used for solving static optimization problems where the underlying system is any discrete-event stochastic system (not necessarily driven by Markov chains). 
This book provides a selected collection of topics, mostly focused on model-free techniques, which are useful when one does not have access to the structure of the objective function (in static optimization) or the transition probability function (in dynamic optimization).   My goal was neither to overwhelm the reader with mathematical details nor was it to cover every topic.  Rather, the goal was to provide the reader with an overview of the fundamental concepts and at the same time provide the details required for solving real-world stochastic optimization problems via simulation-based techniques.
Some of the main topics covered are:

• Reinforcement learning techniques, mainly rooted in Q-Learning for discounted and average reward MDPs
• Static optimization techniques rooted in meta-heuristics (simulated annealing, genetic algorithms, and tabu search) for discrete solution spaces and simultaneous perturbation for continuous solution spaces
• Neural network algorithms useful for function approximation in response surface methods for static optimization and in reinforcement learning for MDPs with large state-action spaces
• A detailed background on dynamic programming (value and policy iteration)
• A special coverage of semi-MDPs (SMDPs) and average reward problems
• A discussion on convergence of a subset of methods enumerated above

From the Back Cover

Simulation-Based Optimization: Parametric Optimization Techniques and Reinforcement Learning introduces the evolving area of static and dynamic simulation-based optimization. Covered in detail are model-free optimization techniques – especially designed for those discrete-event, stochastic systems which can be simulated but whose analytical models are difficult to find in closed mathematical forms.

Key features of this revised and improved Second Edition include:

· Extensive coverage, via step-by-step recipes, of powerful new algorithms for static simulation optimization, including simultaneous perturbation, backtracking adaptive search, and nested partitions, in addition to traditional methods, such as response surfaces, Nelder-Mead search, and meta-heuristics (simulated annealing, tabu search, and genetic algorithms)

· Detailed coverage of the Bellman equation framework for Markov Decision Processes (MDPs), along with dynamic programming (value and policy iteration) for  discounted, average, and total reward performance metrics

· An in-depth consideration of dynamic simulation optimization via temporal differences and Reinforcement Learning: Q-Learning, SARSA, and R-SMART algorithms, and policy search, via API, Q-P-Learning, actor-critics, and learning automata

· A special examination of neural-network-based function approximation for Reinforcement Learning, semi-Markov decision processes (SMDPs), finite-horizon problems, two time scales, case studies for industrial tasks, computer codes (placed online), and convergence proofs, via Banach fixed point theory and Ordinary Differential Equations

Themed around three areas in separate sets of chapters – Static Simulation Optimization, Reinforcement Learning, and Convergence Analysis – this book is written for researchers and students in the fields of engineering (industrial, systems, electrical, and computer), operations research, computer science, and applied mathematics.

Edition details

• Publisher: Springer
• Publication date: 2003
• ISBN 10: 1402074549
• ISBN 13: 9781402074547
• Binding: Hardcover
• Language: English
• Number of pages: 554