Information Theory: An Exploration Across Disciplines

Preface
Acknowledgments
Contents
Abbreviations
1 Introduction, Definitions, and Some General Comments on Shannon’s Measure of Information
	1.1 Definition
	1.2 The Meaning of the Quantity H
	1.3 The Units of Information; the Bit
	1.4 SMI Should not Be Referred to as Entropy
	1.5 Information, Information Theory, and the Monty-Hall Problem
	1.6 “Self-Information” and Probability of Single Event
	1.7 A Comment on Probability and Entropy
	References
2 Information Theory and the Living System
	2.1 Introduction, the Language of the DNA
	2.2 Gatlin’s Book: “Information Theory and the Living System”
	2.3 The “Book of Life” and the “Language of the DNA”
	2.4 Introducing Entropy in the Study of DNA
	2.5 Definitions of Two Derived Quantities: “Divergence from Equiprobability” and “Divergence from Independence”
	2.6 Chapter 3 in Gatlin’s Book is on “Information”
	2.7 Chapter 4 in Gatlin’s Book is on “Living System”
	2.8 Some Recent Works on Applications of Information Theory to Biology
	2.9 Entropy, Information Theory, and Evolution
	2.10 Avery’s Book on Information Theory and Evolution
	2.11 “Genetic Entropy and the Mystery of the Genome” by Sanford (2005) [22]
	References
3 Applications of Information Theory to Psychology
	3.1 Attneave’s Book on Application of Information Theory to Psychology
	3.2 The 20-Questions (20Q) Game and Its Relevance to SMI
	3.3 Redundancy in a Language
	3.4 Chapter 3 of Attneave’s Book is on: “Man\'s Ability to Transmit Information”
	3.5 Earlier Researches on Information Theory and Psychology
	3.6 Some Alternative Experiments Based on the “20Q-Game Psychology”
	References
4 Information Theory and the Arts
	4.1 Introduction
	4.2 Application of SMI to Music
	4.3 Review of Arnheim’s Book on “Entropy and Art”
	References
5 Consistent Definition of Correlations and Multivariate Mutual Information from Information Theory
	5.1 Introduction
	5.2 Binary Correlations
		5.2.1 An Elementary Way to Detect Correlations
		5.2.2 A Second Way to Find Correlations
		5.2.3 Expression of the SMI for US Drivers in 1937
		5.2.4 Searching for the Maximum of SMI
		5.2.5 General Expression for the Uncorrelated Distribution
		5.2.6 Two Exercises on Correlations Between Two Binary Variables
		5.2.7 Correlations When a Variable is Non-Binary
		5.2.8 Measuring the Local and Overall Correlation
		5.2.9 Summary of Section 5.2
	5.3 Ternary Correlations
		5.3.1 Abstract of Section 5.3
		5.3.2 Notations
		5.3.3 Binary and Ternary Correlations for a Trivariate Distribution
		5.3.4 Ternary Correlations for a Trivariate Distribution by Analogy to Binary Correlations
		5.3.5 Constructing the Best 3D Distribution from 2D Marginal Distributions Using the MaxSMI Principle
		5.3.6 An Elementary Example
		5.3.7 The Formula Similar to the Odds Ratio for Ternary Correlations
		5.3.8 An Exercise in the Construction of the Uncorrelated Distribution for Three Binary Variables
		5.3.9 Using the Euler–Lagrange Method
		5.3.10 Iterative Evaluation of Lagrange Multipliers
		5.3.11 The Iterative Proportional Fitting Procedure
		5.3.12 Comparison of the Two Algorithms
		5.3.13 Special Cases: Fixed Elements
		5.3.14 Two Exercises on Completely Uncorrelated Distributions
		5.3.15 Improving the Algorithms to Deal with “Fixed Elements”
		5.3.16 Maximize SMI or Cancel CI
		5.3.17 Summary of Section 5.3
	5.4 Higher-Order Correlations and Correlation Hierarchy
	5.5 Range of the Ternary Correlations
	5.6 Correlations from Different Perspectives
	5.7 General Summary
	5.8 Solutions to the Exercises
	Appendix: The Iterative Evaluation of Lagrange Multipliers Algorithm
	References
6 A Conceptual Consideration of Irreversible Phenomena Based on an Information-Theoretic Measure of Correlations
	6.1 Introduction
	6.2 Reversibility and Irreversibility of Equations and Phenomena
	6.3 Joint and Marginal Probability Distributions/Correlations
	6.4 Uncertainty
	6.5 Uncertainty/SMI Conservation of the Physical System as a Whole
	6.6 Uncertainty (SMI) for a System and Its Subsystems
	6.7 Using SMI to Measure Correlation
	6.8 Correlations Used to Predict Transformations of Two Subsystems A&B
	6.9 Comparison with the Second Law of Thermodynamics
	6.10 The Links Between Correlations and Entropy
	6.11 Irreversibility of a Single System from Statistical Mechanics
	6.12 Irreversibility of a Dilute Gas System from Static Arguments
	6.13 A Simple Reversible Dynamical Model with Irreversible Properties
	6.14 Generalizing the Simple Dynamical Model
	6.15 Synthesis
	6.16 Solutions to the Exercises
	Appendix 1: Proof of the Subadditivity Property
	Appendix 2: Equilibrium PSD from Gibbs Entropy Maximization
	Appendix 3: Equal Entropy and Correlation Increasing Rates for a Dilute Gas
	References
7 Using Mutual Information in Linguistics, Cryptography, and Steganography
	7.1 Summary of this Chapter
	7.2 Applying IT to Pairs of Letters in a Text
	7.3 Applying IT to More Than Two Characters
	7.4 Comparing Two Indicators of Ternary Correlations Between Three Characters
	7.5 More Advanced Comparison Between the Two Indicators of Ternary Correlation
	7.6 Finding Collocations in a Language
	7.7 Application of IT to Phoneme Analysis
	7.8 Mutual Information MI for Three Tokens (Letters, Words, or Phonemes)
	7.9 Using MI in Cryptography and Steganography
		7.9.1 Cryptography and Cryptanalysis
		7.9.2 Steganography
		7.9.3 Detection Techniques for Linguistic Steganography
	7.10 Conclusion
	Appendix: Proof of Three Inequalities on F2sym, F3sym, and MI(3)
	References
8 Applications of SMI to Communication Systems
	8.1 Background
	8.2 The Communication System
	8.3 The Source
	8.4 The Source Coder
		8.4.1 Typical Sequences—Asymptotic Equipartition Principle
		8.4.2 Variable-Length Coding—Practical Source Coders
	8.5 Channel Coding
		8.5.1 Error-Correcting Codes
		8.5.2 Capacity of General Channels
	8.6 A Note on Continuous Signals—Differential SMI
		8.6.1 Efficient Representation of a Continuous Variable
		8.6.2 The Shannon Limit for Continuous Variables
	8.7 AEP and a Postulate About Boltzmann’s Microstates
	8.8 Some Final Words
	Appendix 1: How to Decode the Hamming (7,4) Code
	Appendix 2: Solutions to the Exercises
	References
Index