Recurrent Neural Networks: From Simple to Gated Architectures

This textbook provides a compact but comprehensive treatment that provides analytical and design steps to recurrent neural networks from scratch. It provides a treatment of the general recurrent neural networks with principled methods for training that render the (generalized) backpropagation through time (BPTT).  This author focuses on the basics and nuances of recurrent neural networks, providing technical and principled treatment of the subject, with a view toward using coding and deep learning computational frameworks, e.g., Python and Tensorflow-Keras. Recurrent neural networks are treated holistically from simple to gated architectures, adopting the technical machinery of adaptive non-convex optimization with dynamic constraints to leverage its systematic power in organizing the learning and training processes. This permits the flow of concepts and techniques that provide grounded support for design and training choices. The author’s approach enables strategic co-trainingof output layers, using supervised learning, and hidden layers, using unsupervised learning, to generate more efficient internal representations and accuracy performance. As a result, readers will be enabled to create designs tailoring proficient procedures for recurrent neural networks in their targeted applications.

He was a Visiting Professor at UC, Berkeley (1983), the California Institute of Technology, Pasadena (1992), and the University of Minnesota, Twin Cities (1993). He joined MSU in 1985 and has been a Professor since1991. He has worked and consulted for several companies including General Motors, Ford, Smith’s Industries, Intersignal, IC Tech Inc., and Clarity LLC. He has authored more than 250 technical papers, and co-edited the textbook (Dynamical Systems Approaches to Nonlinear Problems in Circuits and Systems, (SIAM, 1988). He is a co-inventor of more than 14 patents on adaptive nonlinear signal processing, neural networks, and sensors.