Probability, Statistics, and Random Processes for Engineers

Cover......Page 1
Title Page......Page 2
Copyright Page......Page 3
Contents......Page 4
Preface......Page 12
1.1 Introduction: Why Study Probability?......Page 14
Probability as Intuition......Page 15
Probability as the Ratio of Favorable to Total Outcomes (Classical Theory)......Page 16
Probability as a Measure of Frequency of Occurrence......Page 17
Probability Based on an Axiomatic Theory......Page 18
1.3 Misuses, Miscalculations, and Paradoxes in Probability......Page 20
Examples of Sample Spaces......Page 21
1.5 Axiomatic Definition of Probability......Page 28
1.6 Joint, Conditional, and Total Probabilities; Independence......Page 33
Compound Experiments......Page 36
1.7 Bayes’ Theorem and Applications......Page 48
1.8 Combinatorics......Page 51
Occupancy Problems......Page 55
Extensions and Applications......Page 59
1.9 Bernoulli Trials—Binomial and Multinomial Probability Laws......Page 61
Multinomial Probability Law......Page 67
1.10 Asymptotic Behavior of the Binomial Law: The Poisson Law......Page 70
1.11 Normal Approximation to the Binomial Law......Page 76
Summary......Page 78
Problems......Page 79
References......Page 90
2.1 Introduction......Page 92
2.2 Definition of a Random Variable......Page 93
2.3 Cumulative Distribution Function......Page 96
Properties of F[Sub(X)](x)......Page 97
Computation of F[Sub(X)](x)......Page 98
2.4 Probability Density Function (pdf)......Page 101
Four Other Common Density Functions......Page 108
More Advanced Density Functions......Page 110
2.5 Continuous, Discrete, and Mixed Random Variables......Page 113
Some Common Discrete Random Variables......Page 115
2.6 Conditional and Joint Distributions and Densities......Page 120
Properties of Joint CDF F[Sub(XY)](x, y)......Page 131
2.7 Failure Rates......Page 150
Problems......Page 154
Additional Reading......Page 162
3.1 Introduction......Page 164
Functions of a Random Variable (FRV): Several Views......Page 167
3.2 Solving Problems of the Type Y = g(X)......Page 168
General Formula of Determining the pdf of Y = g(X)......Page 179
3.3 Solving Problems of the Type Z = g(X, Y)......Page 184
Fundamental Problem......Page 206
Obtaining fVW Directly from fXY......Page 209
3.5 Additional Examples......Page 213
Summary......Page 218
Problems......Page 219
Additional Reading......Page 227
4.1 Expected Value of a Random Variable......Page 228
On the Validity of Equation 4.1-8......Page 231
4.2 Conditional Expectations......Page 245
Conditional Expectation as a Random Variable......Page 252
4.3 Moments of Random Variables......Page 255
Joint Moments......Page 259
Properties of Uncorrelated Random Variables......Page 261
Jointly Gaussian Random Variables......Page 264
4.4 Chebyshev and Schwarz Inequalities......Page 268
Markov Inequality......Page 270
The Schwarz Inequality......Page 271
4.5 Moment-Generating Functions......Page 274
4.6 Chernoff Bound......Page 277
4.7 Characteristic Functions......Page 279
Joint Characteristic Functions......Page 286
The Central Limit Theorem......Page 289
4.8 Additional Examples......Page 294
Summary......Page 296
Problems......Page 297
References......Page 306
Additional Reading......Page 307
5.1 Joint Distribution and Densities......Page 308
5.2 Multiple Transformation of Random Variables......Page 312
5.3 Ordered Random Variables......Page 315
Distribution of area random variables......Page 318
5.4 Expectation Vectors and Covariance Matrices......Page 324
5.5 Properties of Covariance Matrices......Page 327
Whitening Transformation......Page 331
5.6 The Multidimensional Gaussian (Normal) Law......Page 332
5.7 Characteristic Functions of Random Vectors......Page 341
Properties of CF of Random Vectors......Page 343
The Characteristic Function of the Gaussian (Normal) Law......Page 344
Summary......Page 345
Problems......Page 346
Additional Reading......Page 352
6.1 Introduction......Page 353
Independent, Identically Distributed (i.i.d.) Observations......Page 354
Estimation of Probabilities......Page 356
6.2 Estimators......Page 359
6.3 Estimation of the Mean......Page 361
Properties of the Mean-Estimator Function (MEF)......Page 362
Confidence Interval for the Mean of a Normal Distribution When σ[sub(X)] Is Not Known......Page 365
6.4 Estimation of the Variance and Covariance......Page 368
Confidence Interval for the Variance of a Normal Random variable......Page 370
Estimating the Standard Deviation Directly......Page 372
Estimating the covariance......Page 373
6.5 Simultaneous Estimation of Mean and Variance......Page 374
6.6 Estimation of Non-Gaussian Parameters from Large Samples......Page 376
6.7 Maximum Likelihood Estimators......Page 378
6.8 Ordering, more on Percentiles, Parametric Versus Nonparametric Statistics......Page 382
The Median of a Population Versus Its Mean......Page 384
Parametric versus Nonparametric Statistics......Page 385
Confidence Interval on the Percentile......Page 386
Confidence Interval for the Median When n Is Large......Page 388
6.9 Estimation of Vector Means and Covariance Matrices......Page 389
Estimation of μ......Page 390
Estimation of the covariance K......Page 391
6.10 Linear Estimation of Vector Parameters......Page 393
Problems......Page 397
References......Page 401
Additional Reading......Page 402
7 Statistics: Part 2 Hypothesis Testing......Page 403
7.1 Bayesian Decision Theory......Page 404
7.2 Likelihood Ratio Test......Page 409
7.3 Composite Hypotheses......Page 415
Generalized Likelihood Ratio Test (GLRT)......Page 416
How Do We Test for the Equality of Means of Two Populations?......Page 421
Testing for the Equality of Variances for Normal Populations: The F-test......Page 425
Testing Whether the Variance of a Normal Population Has a Predetermined Value:......Page 429
7.4 Goodness of Fit......Page 430
7.5 Ordering, Percentiles, and Rank......Page 436
How Ordering is Useful in Estimating Percentiles and the Median......Page 438
Confidence Interval for the Median When n Is Large......Page 441
Distribution-free Hypothesis Testing: Testing If Two Population are the Same Using Runs......Page 442
Ranking Test for Sameness of Two Populations......Page 445
Problems......Page 446
References......Page 452
8 Random Sequences......Page 454
8.1 Basic Concepts......Page 455
Infinite-length Bernoulli Trials......Page 460
Continuity of Probability Measure......Page 465
Statistical Specification of a Random Sequence......Page 467
8.2 Basic Principles of Discrete-Time Linear Systems......Page 484
8.3 Random Sequences and Linear Systems......Page 490
8.4 WSS Random Sequences......Page 499
Power Spectral Density......Page 502
Interpretation of the psd......Page 503
Synthesis of Random Sequences and Discrete-Time Simulation......Page 506
Decimation......Page 509
Interpolation......Page 510
8.5 Markov Random Sequences......Page 513
ARMA Models......Page 516
Markov Chains......Page 517
8.6 Vector Random Sequences and State Equations......Page 524
8.7 Convergence of Random Sequences......Page 526
8.8 Laws of Large Numbers......Page 534
Problems......Page 539
References......Page 554
9 Random Processes......Page 556
9.1 Basic Definitions......Page 557
Asynchronous Binary Signaling......Page 561
Poisson Counting Process......Page 563
Alternative Derivation of Poisson Process......Page 568
Random Telegraph Signal......Page 570
Digital Modulation Using Phase-Shift Keying......Page 571
Wiener Process or Brownian Motion......Page 573
Markov Random Processes......Page 576
Birth–Death Markov Chains......Page 580
Chapman–Kolmogorov Equations......Page 584
9.3 Continuous-Time Linear Systems with Random Inputs......Page 585
White Noise......Page 590
9.4 Some Useful Classifications of Random Processes......Page 591
Stationarity......Page 592
9.5 Wide-Sense Stationary Processes and LSI Systems......Page 594
Wide-Sense Stationary Case......Page 595
Power Spectral Density......Page 597
An Interpretation of the psd......Page 599
More on White Noise......Page 603
Stationary Processes and Difierential Equations......Page 609
9.6 Periodic and Cyclostationary Processes......Page 613
9.7 Vector Processes and State Equations......Page 619
State Equations......Page 621
Problems......Page 624
References......Page 646
Stochastic Continuity and Derivatives [10-1]......Page 648
Further Results on m.s. Convergence [10-1]......Page 658
10.2 Mean-Square Stochastic Integrals......Page 663
10.3 Mean-Square Stochastic Di.erential Equations......Page 666
10.4 Ergodicity [10-3]......Page 671
10.5 Karhunen–Loève Expansion [10-5]......Page 678
Bandlimited Processes......Page 684
Bandpass Random Processes......Page 687
WSS Periodic Processes......Page 690
Fourier Series for WSS Processes......Page 693
Appendix: Integral Equations......Page 695
Existence Theorem......Page 696
Problems......Page 699
References......Page 712
11.1 Estimation of Random Variables and Vectors......Page 713
More on the Conditional Mean......Page 719
Orthogonality and Linear Estimation......Page 721
Some Properties of the Operator Ê......Page 729
11.2 Innovation Sequences and Kalman Filtering......Page 731
Predicting Gaussian Random Sequences......Page 735
Kalman Predictor and Filter......Page 737
Error-Covariance Equations......Page 742
11.3 Wiener Filters for Random Sequences......Page 746
Unrealizable Case (Smoothing)......Page 747
Causal Wiener Filter......Page 749
11.4 Expectation-Maximization Algorithm......Page 751
Log-likelihood for the Linear Transformation......Page 753
Summary of the E-M algorithm......Page 755
E-M Algorithm for Exponential Probability Functions......Page 756
Application to Emission Tomography......Page 757
Log-likelihood Function of Complete Data......Page 759
E-step......Page 760
M-step......Page 761
11.5 Hidden Markov Models (HMM)......Page 762
Speci.cation of an HMM......Page 764
Application to Speech Processing......Page 766
E.cient Computation of P[E|M] with a Recursive Algorithm......Page 767
Viterbi Algorithm and the Most Likely State Sequence for the Observations......Page 769
11.6 Spectral Estimation......Page 772
The Periodogram......Page 773
Bartlett’s Procedure–Averaging Periodograms......Page 775
Parametric Spectral Estimate......Page 780
Maximum Entropy Spectral Density......Page 782
11.7 Simulated Annealing......Page 785
Gibbs Sampler......Page 786
Noncausal Gauss–Markov Models......Page 787
Compound Markov Models......Page 791
Gibbs Line Sequence......Page 792
Problems......Page 796
References......Page 801
Sequences......Page 803
Convergence......Page 804
Z-Transform......Page 805
A.2 Continuous Mathematics......Page 806
Definite and Indefinite Integrals......Page 807
Difierentiation of Integrals......Page 808
Completing the Square......Page 809
Functions......Page 810
A.3 Residue Method for Inverse Fourier Transformation......Page 812
Fact......Page 813
Inverse Fourier Transform for psd of Random Sequence......Page 815
References......Page 819
B.1 Gamma Function......Page 820
B.3 Dirac Delta Function......Page 821
References......Page 824
C.1 Introduction......Page 825
C.2 Jacobians for n = 2......Page 826
C.3 Jacobian for General n......Page 828
D.1 Introduction and Basic Ideas......Page 831
D.2 Application of Measure Theory to Probability......Page 833
Distribution Measure......Page 834
Appendix E: Sampled Analog Waveforms and Discrete-time Signals......Page 835
Appendix F: Independence of Sample Mean and Variance for Normal Random Variables......Page 837
Appendix G: Tables of Cumulative Distribution Functions: the Normal, Student t, Chi-square, and F......Page 840
B......Page 848
C......Page 849
D......Page 850
F......Page 851
I......Page 852
L......Page 853
M......Page 854
O......Page 855
P......Page 856
S......Page 857
U......Page 858
Z......Page 859