Probability and Random Processes With Applications to Signal Processing

Provides users with an accessible, yet mathematically solid, treatment of probability and random processes. Many computer examples integrated throughout, including random process examples in MATLAB. Includes expanded discussions of fundamental principles, especially basic probability. Includes new problems which deal with applications of basic theory—in such areas as medical imaging, percolation theory in fractals, and generation of random numbers. Several new topics include Failure rates, the Chernoff bound, interval estimation and the Student t-distribution, and power spectral density estimation. Functions of Random Variables is included as a separate chapter. Mean square convergence and introduction of Martingales is covered in the latter half of the book. Provides electrical and computer engineers with a solid treatment of probability and random processes.

Preface

The first edition of this book (1986) grew out of a set of notes used by the authors to teach two one-semester courses on probability and random processes at Rensselaer Polytechnic Institute (RPI). At that time the probability course at RPI was required of all students in the Computer and Systems Engineering Program and was a highly recommended elective for students in closely related areas. While many undergraduate students took the course in the junior year, many seniors and first-year graduate students took the course for credit as well. Then, as now, most of the students were engineering students. To serve these students well, we felt that we should be rigorous in introducing fundamental principles while furnishing many opportunities for students to develop their skills at solving problems.

There are many books in this area and they range widely in their coverage and depth. At one extreme are the very rigorous and authoritative books that view probability from the point of view of measure theory and relate probability to rather exotic theorems such as the Radon-Nikodym theorem (see for example Probability and Measure by Patrick Billingsley, Wiley, 1978). At the other extreme are books that usually combine probability and statistics and largely omit underlying theory and the more advanced types of applications of probability. In the middle are the large number of books that combine probability and random processes, largely avoiding a measure theoretic approach, preferring to emphasize the axioms upon which the theory is based. It would be fair to say that our book falls into this latter category. Nevertheless this begs the question: why write or revise another book in this area if there are already several good texts out there that use the same approach and provide roughly the same coverage? Of course back in 1986 there were few books that emphasized the engineering applications of probability and random processes and that integrated the latter into one volume. Now there are several such books.

Both authors have been associated (both as students and faculty) with colleges and universities that have demanding programs in engineering and applied science. Thus their experience and exposure have been to superior students that would not be content with a text that furnished a shallow discussion of probability. At the same time, however, the authors wanted to write a book on probability and random processes for engineering and, applied science students. A measure-theoretic book, or one that avoided the engineering applications of probability and the processing of random signals, was regarded not suitable for such students. At the same time the authors felt that the book should have enough depth so that students taking 2nd year graduate courses in advanced topics such as estimation and detection, pattern recognition, voice and image processing, networking and queuing, and so forth would not be handicapped by insufficient knowledge of the fundamentals and applications of random phenomena. In a nutshell we tried to write a book that combined rigor with accessibility and had a strong self-teaching orientation. To that end we included a. large number of worked-out examples, MATLAB codes, and special appendices that include a review of the kind of basic math needed for solving problems in probability as well as an introduction to measure theory and its relation to probability. The MATLAB codes, as well as other useful material such as multiple choice exams that cover each of the book's sections, can be found at the book's web site http://www.prenhall.com/stark.

The normal use of this book would be as follows: for a first course in probability at, say the junior or senior year, a reasonable goal is to cover Chapters 1 through 4. Nevertheless we have found that this may be too much for students not well prepared in mathematics. In that case we suggest a load reduction in which combinatorics in Chapter 1 (parts of Section 1.8), failure rates in Chapter 2 (Section 2.7), more advanced density functions and the Poisson transform in Chapter 3 are lightly or not covered the first time around. The proof of the Central Limit Theorem, joint characteristic functions, and Section 4.8, which deals with statistics, all in Chapter 4, can, likewise, also be omitted on a first reading.

Chapters 5 to 9 provide the material for a first course in random processes. Normally such a course is taken in the first year of graduate studies and is required for all further study in signal processing, communications, computer and communication networking, controls, and estimation and detection theory. Here what to cover is given greater latitude. If pressed for time, we suggest that the pattern recognition applications and simultaneous diagonalization of two covariance matrices in Chapter 5 be given lower preference than the other material in that chapter. Chapters 6 and 7 are essential for any course in random processes and the coverage of the topics therein should be given high priority. Chapter 9 on signal processing should, likewise be given high priority, because it illustrates the applications of the theory to current state-of-art problems. However, within Chapter 9, the instructor can choose among a number of applications and need not cover them all if time pressure becomes an issue. Chapter 8 dealing with advanced topics is critically important to the more advanced students, especially those seeking further studies toward the Ph.D. Nevertheless it too can be lightly covered or omitted in a first course if time is the critical factor.

Readers familiar with the 2nd edition of this book will find significant changes in the 3rd edition. The changes were the result of numerous suggestions made by lecturers and students alike. To begin with, we modified the title to Probability and Random Processes with Applications to Signal Processing, to better reflect the contents. We removed the two chapters on estimation theory and moved some of this material to other chapters where it naturally fitted in with the material already there. Some of the material on parameter estimation e.g., the Gauss-Markov Theorem has been removed, owing to the need for finding space for new material. In terms of organization, the major changes have been in the random processes part of the book. Many readers preferred seeing discrete-time random phenomena in one chapter and continuous-time phenomena in another chapter. In the earlier editions of the book there was a division along these lines but also a secondary division along the lines of stationary versus non-stationary processes. For some this made the book awkward to teach from. Now all of the material on discrete-time phenomena appears in one chapter (Chapter 6); likewise for continuous-time phenomena (Chapter 7). Another major change is a new Chapter 9 that discusses applications to signal processing. Included are such topics as: the orthogonality principle, Wiener and Kalman filters, The Expectation-Maximization algorithm, Hidden Markov Models, and simulated annealing. Chapter 8 (Advanced Topics) covers much of the same ground as the old Chapter 9 e.g., stochastic continuity, meansquare convergence, Ergodicity etc. and material from the old Chapter 10 on representation of random processes.

There have been significant changes in the first half of the book also. For example, in Chapter 1 there is an added section on the misuses of probability in ordinary life. Here we were helped by the discussions in Steve Pinker's excellent book How the Mind Works (Norton Publishers, New York, 1997). Chapter 2 (Random Variables) now includes discussions 6n more advanced distributions such as the Gamma, Chi-square and the Student-t. All of the chapters have many more worked-out examples as well as more homework problems. Whenever convenient we tried to use MATLAB to obtain graphical results. Also, being a book primarily for engineers, many of the worked-out example and homework problems relate to real-life systems or situations.

We have added several new appendices to provide the necessary background mathematics for certain results in the text and to enrich the reader's understanding of probability. An appendix on Measure Theory falls in the latter category. Among the former are appendices on the delta and gamma functions, probability-related basic math, including the principle of proof-by-induction, Jacobians for n-dimensional transformations, and material on Fourier and Laplace inversion.

For this edition, the authors would like to thank Geoffrey Williamson and Yongyi Yang for numerous insightful discussions and help with some of the MATLAB programs. Also we thank Nikos Galatsanos, Miles Wernick, Geoffrey Chan, Joseph LoCicero, and Don Ucci for helpful suggestions. We also would like to thank the administrations of Illinois Institute of Technology and Rensselaer Polytechnic Institute for their patience and support while this third edition was being prepared. Of course, in the end, it is the reaction of the students that is the strongest driving force for improvements. To all our students and readers we owe a large debt of gratitude.

Henry Stark
John W. Woods