Computational Intelligence: A Methodological Introduction

Preface
Contents
1 Introduction
	1.1 Intelligent Systems
	1.2 Computational Intelligence
	1.3 About the Third Edition of this Book
Part I Neural Networks
2 Introduction to Artificial Neural Networks
	2.1 Motivation
	2.2 Biological Background
3 Threshold Logic Units
	3.1 Definition and Examples
	3.2 Geometric Interpretation
	3.3 Limitations
	3.4 Networks of Threshold Logic Units
	3.5 Training the Parameters
	3.6 Variants
	3.7 Training Networks
4 General Neural Networks
	4.1 Structure of Neural Networks
	4.2 Operation of Neural Networks
	4.3 Scale Types and Encoding
	4.4 Training Neural Networks
5 Multi-layer Perceptrons
	5.1 Definition and Examples
	5.2 Why Non-linear Activation Functions?
	5.3 Function Approximation
	5.4 Logistic Regression
	5.5 Gradient Descent
	5.6 Error Backpropagation
	5.7 Gradient Descent Examples
	5.8 Variants of Gradient Descent
		5.8.1 Manhattan Training
		5.8.2 Momentum Term
		5.8.3 Nesterov\'s Accelerated Gradient (NAG)
		5.8.4 Self-adaptive Error Backpropagation (SuperSAB)
		5.8.5 Resilient Backpropagation
		5.8.6 Quick Propagation
		5.8.7 Adaptive Subgradient Descent (AdaGrad)
		5.8.8 Root Mean Squared Gradient Descent (RMSProp)
		5.8.9 Adaptive Subgradient Descent Over Windows (AdaDelta)
		5.8.10 Adaptive Moment Estimation (Adam)
		5.8.11 Lifting the Derivative of the Activation Function
		5.8.12 Weight Decay
		5.8.13 Batch Normalization
	5.9 Examples for Some Variants
	5.10 Initializing the Parameters
	5.11 Number of Hidden Neurons
	5.12 Deep Learning
	5.13 Sensitivity Analysis
6 Radial Basis Function Networks
	6.1 Definition and Examples
	6.2 Function Approximation
	6.3 Initializing the Parameters
	6.4 Training the Parameters
	6.5 Example of Training
	6.6 Generalized Form
7 Self-organizing Maps
	7.1 Definition and Examples
	7.2 Learning Vector Quantization
	7.3 Neighborhood of the Output Neurons
8 Hopfield Networks
	8.1 Definition and Examples
	8.2 Convergence of the Computations
	8.3 Associative Memory
	8.4 Solving Optimization Problems
	8.5 Simulated Annealing
	8.6 Boltzmann Machines
9 Recurrent Networks
	9.1 Simple Examples
	9.2 Representing Differential Equations
	9.3 Vectorial Neural Networks
	9.4 Error Backpropagation Through Time
	9.5 Long Short-Term Memory
10 Neural Networks: Mathematical Remarks
	10.1 Equations for Straight Lines
	10.2 Regression
	10.3 Activation Transformation
Part II Evolutionary Algorithms
11 Introduction to Evolutionary Algorithms
	11.1 Metaheuristics
	11.2 Biological Evolution
	11.3 Simulated Evolution
		11.3.1 Optimization Problems
		11.3.2 Basic Notions and Concepts
		11.3.3 Building Blocks of an Evolutionary Algorithm
	11.4 The n-Queens Problem
	11.5 Related Optimization Techniques
		11.5.1 Gradient Ascent or Descent
		11.5.2 Hill Climbing
		11.5.3 Simulated Annealing
		11.5.4 Threshold Accepting
		11.5.5 Great Deluge Algorithm
		11.5.6 Record-to-Record Travel
	11.6 The Traveling Salesman Problem
12 Elements of Evolutionary Algorithms
	12.1 Encoding of Solution Candidates
		12.1.1 Hamming Cliffs
		12.1.2 Epistasis
		12.1.3 Closedness of the Search Space
	12.2 Fitness and Selection
		12.2.1 Fitness Proportionate Selection
		12.2.2 The Dominance Problem
		12.2.3 Vanishing Selective Pressure
		12.2.4 Adapting the Fitness Function
		12.2.5 The Variance Problem
		12.2.6 Rank-Based Selection
		12.2.7 Tournament Selection
		12.2.8 Elitism
		12.2.9 Environmental Selection
		12.2.10 Niche Techniques
		12.2.11 Characterization of Selection Methods
	12.3 Genetic Operators
		12.3.1 Mutation Operators
		12.3.2 Crossover Operators
		12.3.3 Multi-parent Operators
		12.3.4 Characteristics of Recombination Operators
		12.3.5 Interpolating and Extrapolating Recombination
13 Fundamental Evolutionary Algorithms
	13.1 Genetic Algorithms
		13.1.1 The Schema Theorem
		13.1.2 The Two-Armed Bandit Argument
		13.1.3 The Principle of Minimal Alphabets
	13.2 Evolution Strategies
		13.2.1 Selection
		13.2.2 Global Variance Adaptation
		13.2.3 Local Variance Adaptation
		13.2.4 Covariances
		13.2.5 Recombination Operators
	13.3 Genetic Programming
		13.3.1 Initialization
		13.3.2 Genetic Operators
		13.3.3 Application Examples
		13.3.4 The Problem of Introns
		13.3.5 Extensions
	13.4 Multi-objective Optimization
		13.4.1 Weighted Combination of Objectives
		13.4.2 Pareto-Optimal Solutions
		13.4.3 Finding Pareto Frontiers with Evolutionary Algorithms
	13.5 Special Applications and Techniques
		13.5.1 Behavioral Simulation
		13.5.2 Parallelization
14 Computational Swarm Intelligence
	14.1 Introduction
	14.2 Basic Principles of Computational Swarm Intelligence
		14.2.1 Swarms in Known Environments
		14.2.2 Swarms in Unknown Environments
	14.3 Particle Swarm Optimization
		14.3.1 Influence of the Parameters
		14.3.2 Turbulence Factor
		14.3.3 Boundary Handling
	14.4 Multi-objective Particle Swarm Optimization
		14.4.1 Leader Selection Mechanism
		14.4.2 Archiving
	14.5 Many-objective Particle Swarm Optimization
		14.5.1 Ranking Non-dominated Solutions
		14.5.2 Distance Based Ranking
	14.6 Ant Colony Optimization
Part III Fuzzy Systems
15 Introduction to Fuzzy Sets and Fuzzy Logics
	15.1 Natural Languages and Formal Models
	15.2 Fuzzy Sets
	15.3 Representation of Fuzzy Sets
		15.3.1 Definition Based on Functions
		15.3.2 α-Cuts
	15.4 Fuzzy Logic
		15.4.1 Propositions and Truth Values
		15.4.2 t-Norms and t-Conorms
		15.4.3 Aggregation Functions
	15.5 Semantics of Membership Degrees
		15.5.1 Membership Degrees as Truth Degrees
		15.5.2 Membership Degrees as Similarity to a Reference Value
		15.5.3 Membership Degrees as Preferences
		15.5.4 Membership Degrees as Possibility
		15.5.5 Consistent Interpretations of Fuzzy Sets in Applications
	15.6 Operations on Fuzzy Sets
		15.6.1 Intersection
		15.6.2 Union
		15.6.3 Complement
		15.6.4 Covering and Partition
		15.6.5 Linguistic Modifiers
	15.7 Fuzzy Sets of Type 2
16 The Extension Principle
	16.1 Mappings of Fuzzy Sets
	16.2 Mappings of a-cuts
	16.3 Cartesian Product and Cylindrical Extension
	16.4 Extension Principle for Multivariate Mappings
17 Fuzzy Relations
	17.1 Crisp Relations
	17.2 Application of Relations
	17.3 Logical Deduction with Relations
	17.4 Simple Fuzzy Relations
	17.5 Composition of Fuzzy Relations
	17.6 Fuzzy Relational Equations
18 Similarity Relations
	18.1 Similarity
	18.2 Fuzzy Sets and Extensional Hulls
	18.3 Scaling Concepts
	18.4 Fuzzy Sets and Similarity Relations
19 Approximate Reasoning
	19.1 Linguistic Variables
	19.2 Computing with Words
	19.3 Generalized Logical Inference
	19.4 Approximation of Functions Using Linguistic If-Then Rules
		19.4.1 Approximation of Functions by Using Rules as Constraints
		19.4.2 Approximation of Functions by Solving Fuzzy Relational Equations
		19.4.3 Approximation of Functions by Interpolation Between Fuzzy Points
20 Fuzzy Control
	20.1 Mamdani Fuzzy Controller
	20.2 Design of a Mamdani Fuzzy Controller
	20.3 Mamdani Controller and Similarity Relations
		20.3.1 Interpretation of a Mamdani Controller Using Similarity Relations
		20.3.2 Construction of a Mamdani Controller Using Similarity Relations
	20.4 Takagi–Sugeno Controller
21 Hybrid Systems for Tuning and Learning Fuzzy Systems
	21.1 Neuro-Fuzzy Control
		21.1.1 Models with Supervised Learning Methods
		21.1.2 Models with Reinforcement Learning
	21.2 Evolutionary Fuzzy Control
		21.2.1 Structure of an Evolutionary Fuzzy Controller
		21.2.2 Optimizing Parameters of an Evolutionary Fuzzy Controller
		21.2.3 Example
22 Fuzzy Data Analysis
	22.1 Fuzzy Methods in Data Analysis
	22.2 Fuzzy Clustering
		22.2.1 Clustering
		22.2.2 Presuppositions and Notation
		22.2.3 Classical c-Means Clustering
		22.2.4 Fuzzification by Membership Transformation
		22.2.5 Fuzzification by Membership Regularization
		22.2.6 Comparison
	22.3 Analysis of Precise Data with Possibility Theory
	22.4 Analysis of Imprecise Data Using Random Sets
	22.5 Analysis of Fuzzy Data with Fuzzy Random Variables
Part IV Bayes and Markov Networks
23 Bayesian Networks
24 Elements of Probability and Graph Theory
	24.1 Probability Theory
		24.1.1 Random Variables and Random Vectors
		24.1.2 Independences
	24.2 Graph Theory
		24.2.1 Background
		24.2.2 Join Graphs
		24.2.3 Separations
25 Decompositions
26 Evidence Propagation
	26.1 Initialization
	26.2 Message Passing
	26.3 Update
	26.4 Marginalization
27 Learning Graphical Models
	27.1 Score-Based Approaches
		27.1.1 Likelihood of a Database
		27.1.2 K2 Algorithm
	27.2 Constraint-Based Approaches
28 Belief Revision
	28.1 Introduction
	28.2 Revision Procedure
	28.3 A Real-World Application
29 Decision Graphs
	29.1 Motivation
	29.2 Definition
	29.3 Policies and Strategies
	29.4 Finding Optimal Strategies
	29.5 Example Scenario
30 Causal Networks
	30.1 Causal and Probabilistic Structure
	30.2 The Do Operator
Index