Entropy and information theory

14.2 Feinstein’s Lemma......Page 3
 8.3 A Mean Ergodic Theorem for Densities......Page 7
 12.3 Block Source Code Mismatch......Page 13
 14.6 Block Coding Theorems for Noisy Channels......Page 17
Cover......Page 1
  Reference Letters......Page 5
  Isomorphic Random Processes......Page 12
  Process approximation......Page 16
 12.2 Block Source Codes for AMS Sources......Page 2
 14.4 Channel Capacity......Page 9
Index......Page 10
  Finite Alphabets......Page 11
  Variational Description of Divergence......Page 15
 Introduction......Page 18
 14.8 Synchronizing Block Channel Codes......Page 22
 6.7 Modeling and Simulation......Page 23
 12.6 Subadditive Fidelity Criteria......Page 25
  Totally Ergodic Sources......Page 26
  Generalized Conditional Relative Entropy......Page 28
 12.8 Sliding-Block Source Codes......Page 29
 1.1 Probability Spaces and Random Variables......Page 30
 3.8 Markov Approximations......Page 31
 7.4 Information for General Alphabets......Page 32
  Ergodic Sources......Page 33
  Partitions......Page 35
 1.3 Distributions......Page 36
  Strengthened Rohlin-Kakutani Theorem......Page 37
 Contents......Page 14
  The dp-distance......Page 21
Entropy and Information Theory......Page 4
 Preface......Page 8
 14.7 Joint Source and Channel Block Codes......Page 19
 14.3 Feinstein’s Theorem......Page 6
 7.5 Convergence Results......Page 44
 12.7 Asynchronous Block Codes......Page 27
 12.9 A Geometric Interpretation......Page 39
  Finite-order distribution Shannon conditions for IID processes......Page 20
 1.2 Random Processes and Dynamical Systems......Page 34
 1.4 Standard Alphabets......Page 41
 1.5 Expectation......Page 42
 1.6 Asymptotic Mean Stationarity......Page 45
 1.7 Ergodic Properties......Page 46
 2.1 Pair Processes......Page 49
 2.2 Channels......Page 50
 2.3 Stationarity Properties of Channels......Page 53
  Completely Random Channels......Page 57
 2.5 Deterministic Channels and Sequence Coders......Page 58
 2.6 Stationary and Sliding-Block Codes......Page 59
  Finite-length Sliding-Block Codes......Page 61
  B-Processes......Page 63
  Block Independent Processes......Page 65
 2.8 Random Punctuation Sequences......Page 66
 2.10 Finite-Memory Channels......Page 70
 2.11 Output Mixing Channels......Page 71
 2.12 Block Independent Channels......Page 73
 2.13 Conditionally Block Independent Channels......Page 74
 2.15 Primitive Channels......Page 76
 2.17 Markov Channels......Page 77
 2.18 Finite-State Channels and Codes......Page 78
 2.19 Cascade Channels......Page 79
 2.20 Communication Systems......Page 80
 2.22 Block to Sliding-Block: The Rohlin-Kakutani Theorem......Page 81
  Partitions......Page 83
  Gadgets......Page 84
  Strengthened Rohlin-Kakutani Theorem......Page 85
 3.1 Entropy and Entropy Rate......Page 89
 3.2 Divergence Inequality and Relative Entropy......Page 93
 3.3 Basic Properties of Entropy......Page 97
  Concavity of Entropy......Page 100
  Entropy and Binomial Sums......Page 102
  Variational Description of Divergence......Page 104
 3.4 Entropy Rate......Page 106
 3.5 Relative Entropy Rate......Page 109
 3.6 Conditional Entropy and Mutual Information......Page 110
 3.7 Entropy Rate Revisited......Page 118
 3.8 Markov Approximations......Page 119
 3.9 Relative Entropy Densities......Page 121
 4.1 History......Page 124
 4.2 Stationary Ergodic Sources......Page 127
 4.3 Stationary Nonergodic Sources......Page 133
 4.4 AMS Sources......Page 137
 4.5 The Asymptotic Equipartition Property......Page 141
 5.1 Distortion Measures......Page 143
 5.2 Fidelity Criteria......Page 146
 5.3 Average Limiting Distortion......Page 147
 5.4 Communications Systems Performance......Page 149
 5.5 Optimal Performance......Page 150
 5.7 Approximating Random Vectors and Processes......Page 155
 5.8 The Monge/Kantorovich/Vasershtein Distance......Page 158
 5.10 Coupling Discrete Spaces with the Hamming Distance......Page 160
 5.11 Process Distance and Approximation......Page 161
  The dp-distance......Page 163
  Evaluating Process Distortion......Page 167
 5.13 d-bar Continuous Channels......Page 168
 6.1 The Fano Inequality......Page 173
 6.2 Code Approximation and Entropy Rate......Page 176
  Dynamical Systems and Random Processes......Page 177
 6.3 Pinsker’s and Marton’s Inequalities......Page 178
 6.4 Entropy and Isomorphism......Page 182
  Isomorphic Probability Spaces......Page 183
  Isomorphism Mod 0......Page 184
 6.5 Almost Lossless Source Coding......Page 186
  Almost-Lossless Block Codes......Page 187
  Asynchronous Block Code......Page 189
  Sliding-Block Code......Page 191
 6.6 Asymptotically Optimal Almost Lossless Codes......Page 194
 6.7 Modeling and Simulation......Page 195
 7.1 Divergence......Page 198
  Variational Description of Divergence......Page 212
 7.2 Conditional Relative Entropy......Page 214
  Generalized Conditional Relative Entropy......Page 225
 7.3 Limiting Entropy Densities......Page 227
 7.4 Information for General Alphabets......Page 229
 7.5 Convergence Results......Page 241
 8.1 Information Rates for Finite Alphabets......Page 244
 8.2 Information Rates for General Alphabets......Page 246
 8.3 A Mean Ergodic Theorem for Densities......Page 250
 8.4 Information Rates of Stationary Processes......Page 252
 8.5 The Data Processing Theorem......Page 259
 8.6 Memoryless Channels and Sources......Page 260
 9.1 The Shannon Distortion-Rate Function......Page 262
 9.2 Basic Properties......Page 264
  IID Sources......Page 266
 9.3 Process Definitions of the Distortion-Rate Function......Page 267
 9.4 The Distortion-Rate Function as a Lower Bound......Page 275
 9.5 Evaluating the Rate-Distortion Function......Page 277
  Support of Shannon Optimal Distributions......Page 287
 10.1 Relative Entropy Densities and Rates......Page 289
 10.2 Markov Dominating Measures......Page 292
 10.3 Stationary Processes......Page 296
  Finite Alphabets......Page 299
  Standard Alphabets......Page 302
 11.1 Stationary Ergodic Sources......Page 304
 11.2 Stationary Nonergodic Sources......Page 309
 11.3 AMS Sources......Page 313
 11.4 Ergodic Theorems for Information Densities.......Page 316
 12.1 Source Coding and Channel Coding......Page 318
 12.2 Block Source Codes for AMS Sources......Page 319
  Reference Letters......Page 322
  Performance and Distortion-Rate Functions......Page 323
 12.3 Block Source Code Mismatch......Page 330
 12.4 Block Coding Stationary Sources......Page 333
 12.5 Block Coding AMS Ergodic Sources......Page 335
 12.6 Subadditive Fidelity Criteria......Page 342
 12.7 Asynchronous Block Codes......Page 344
 12.8 Sliding-Block Source Codes......Page 346
 12.9 A Geometric Interpretation......Page 356
 13.1 Optimal and Asymptotically Optimal Codes......Page 358
 13.2 Block Codes......Page 360
  Moment Properties......Page 365
 13.3 Sliding-Block Codes......Page 366
  Process approximation......Page 373
  Moment conditions......Page 375
  Finite-order distribution Shannon conditions for IID processes......Page 377
  Asymptotic Uncorrelation......Page 379
 14.1 Noisy Channels......Page 381
 14.2 Feinstein’s Lemma......Page 383
 14.3 Feinstein’s Theorem......Page 386
 14.4 Channel Capacity......Page 389
 14.5 Robust Block Codes......Page 394
 14.6 Block Coding Theorems for Noisy Channels......Page 397
 14.7 Joint Source and Channel Block Codes......Page 399
 14.8 Synchronizing Block Channel Codes......Page 402
  Totally Ergodic Sources......Page 406
  Ergodic Sources......Page 413
References......Page 417
Index......Page 426