Computational Optimization. Success in Practice

Cover
Half Title
Series Page
Title Page
Copyright Page
Dedication
Contents
Preface
Author
Acronyms and Abbreviations
List of Algorithms
1. Introduction to Optimization
	1.1. Optimization Models
	1.2. General Notations for Optimization Problem
	1.3. Data Fitting Examples
	1.4. Optimization Fundamentals
		1.4.1. Feasibility
		1.4.2. Optimality
		1.4.3. Convexity
	1.5. General Optimization Algorithm
		1.5.1. Solving Optimization Problems in Iterations
		1.5.2. Termination Criteria
	1.6. Convergence
	1.7. Homework Problems
2. Minimization Approaches for Functions of One Variable
	2.1. Minimizing a Function in 1D
	2.2. Bisection Method
	2.3. Golden Section Search
	2.4. Newton's Method
	2.5. Brute-Force Search
	2.6. Monte Carlo Method
	2.7. Practical Examples
	2.8. Computational Analysis for Convergence
	2.9. Homework Problems
	2.10. Lab Assignment #1: Review Chapters 1–2
3. Generalized Optimization Framework
	3.1. Parameter Identification for Least-Square Data Fitting
	3.2. Generalized Optimization Framework
		3.2.1. Computational Components
		3.2.2. Choice of Proper Software
	3.3. Choosing and Adjusting Optimization Algorithms
	3.4. Visualization and Analysis of Obtained Solutions
	3.5. Testing and Dealing with Problems (Debugging)
	3.6. TEST Mode for the Gradient-based Framework
	3.7. Accuracy and Performance
	3.8. Communication within the Framework
	3.9. Homework Problems
4. Exploring Optimization Algorithms
	4.1. Iterative Optimization Algorithms Revisited
	4.2. Gradient-based Strategies: Line Search vs. Trust Region
		4.2.1. Line Search
		4.2.2. Trust Region
		4.2.3. Comparing Strategies
	4.3. Heuristic Algorithms
	4.4. Particle Swarm Optimization
	4.5. Overview of Optimization Algorithms
	4.6. Homework Problems
5. Line Search Algorithms
	5.1. Local and Global Minimums: Theory Review
		5.1.1. Necessary Conditions
		5.1.2. Necessary vs. Sufficient Conditions
		5.1.3. Existence of Global Minimums
	5.2. Selecting Search Direction
		5.2.1. Principal Generalization
		5.2.2. Some Theory for Convergence
		5.2.3. Steepest Descent
	5.3. Newton's Method and Newton-based Modifications
		5.3.1. Pure Newton's Method
		5.3.2. Discretized Newton's Method
		5.3.3. Modified Newton's Method
		5.3.4. Diagonally Scaled SD
		5.3.5. Quasi-Newton Methods
		5.3.6. Gauss-Newton Approach
	5.4. Conjugate Gradient
	5.5. Line Search Performance Comparison
	5.6. Homework Problems
	5.7. Lab Assignment #2: Review Chapters 3–5
6. Choosing Optimal Step Size
	6.1. Overview
	6.2. Simple Approaches
	6.3. Inexact Line Search
		6.3.1. Armijo Rule
		6.3.2. Wolfe Conditions
		6.3.3. Goldstein Conditions
		6.3.4. Backtracking Line Search
	6.4. Step Size Performance Comparison
	6.5. Advanced Methods for 1D Search
	6.6. Brent's Method
	6.7. Bracketing-Brent Toolbox in MATLAB
		6.7.1. General Description
		6.7.2. Initial Bracketing with fn_min_brack.m
		6.7.3. Brent Minimization with fn_brent.m
		6.7.4. Technicalities for MATLAB Implementation
	6.8. Bracketing-Brent Toolbox Performance
	6.9. Homework Problems
7. Trust Region and Derivative-Free Methods
	7.1. Trust Region Outline
	7.2. General Algorithm
	7.3. Cauchy Point Calculation
	7.4. Improving Cauchy Point by Dogleg Method
	7.5. Checking and Tuning Performance
	7.6. Exploring Derivative-Free Options
	7.7. Homework Problems
	7.8. Lab Assignment #3: Review Chapters 6–7
	7.9. Midterm Assignment: Review Chapters 1–7
8. Large-Scale and Constrained Optimization
	8.1. Generalization of Large-Scale Optimization
	8.2. LSO Examples
		8.2.1. Solving Systems of Equations
		8.2.2. Space-Dependent Parameter Reconstruction
		8.2.3. Parameter Identification – Another Example
		8.2.4. State-Dependent Parameter Reconstruction
	8.3. Improving Performance of LSO
	8.4. General Theory for Constrained Optimization
	8.5. Lagrange Multiplier Approach
	8.6. Lagrange Multiplier vs. Penalization
	8.7. Extending Complexity – DE-constrained Optimization
	8.8. KKT Optimality Conditions
	8.9. Homework Problems
9. ODE-based Optimization
	9.1. Fitting Data by ODE-based Optimization
	9.2. Derivative vs. Directional Derivative: Review
	9.3. Optimize–then–Discretize
		9.3.1. Deriving Gradient
		9.3.2. Solving Problem
	9.4. Discretize–then–Optimize
	9.5. Numerical ODE Solvers
		9.5.1. Runge–Kutta Integration Methods
		9.5.2. Solving ODEs by MATLAB
	9.6. Dynamics of Biological Systems by Lotka–Volterra Equations
	9.7. Optimization Problem Constrained by LV Model
		9.7.1. Statement of Problem
		9.7.2. Deriving Gradients
		9.7.3. Optimization Algorithm
	9.8. Optimization Framework in MATLAB
		9.8.1. Benchmark Models for Controls
		9.8.2. Analytic vs. Synthetic Measurements
		9.8.3. Adjusting Framework and Choosing Parameters
		9.8.4. Checking and Improving Quality of Discretized Gradients
		9.8.5. Optimization Results
	9.9. Homework Problems
10. Implementing Regularization Techniques
	10.1. Motivation for Regularization
	10.2. Some Regularization Theory
		10.2.1. Quadratic Penalty Function Method
		10.2.2. Barrier Functions Method
		10.2.3. Tikhonov-type Regularization
		10.2.4. Gradient Preconditioning
		10.2.5. Bounds by Simple Projections and Slack Variables
	10.3. Examples of Numerical Implementation
		10.3.1. From Theory to Practice
		10.3.2. Adjusting Framework
		10.3.3. Results and Food for Thought
	10.4. Homework Problems
	10.5. Lab Assignment #4: Review Chapters 8–10
11. Moving to PDE-based Optimization
	11.1. Generalized Problem of Fitting Data
	11.2. Practice Example
		11.2.1. Elliptic Equation as Governing PDE
		11.2.2. Deriving Gradient by Optimize–then–Discretize
		11.2.3. Optimization Algorithm
	11.3. Solving PDEs in Higher Dimensions by FreeFEM
	11.4. Brief Introduction to Finite Elements Method
	11.5. Solving PDEs by FreeFEM
		11.5.1. Poisson's Equation in 2D
		11.5.2. Coding with FreeFEM
		11.5.3. Solution Analysis
		11.5.4. Technicalities for FreeFEM Coding
	11.6. Homework Problems
12. Sharing Multiple Software Environments
	12.1. Practice Example and Benchmark Model
	12.2. Updating and Tuning Optimization Framework
		12.2.1. Overview of Changes
		12.2.2. Software Communication
		12.2.3. Another Round on Measurements
		12.2.4. Evaluating Objectives
		12.2.5. Evaluating Gradients
		12.2.6. Checking Quality of Discretized Gradients
	12.3. Analyzing Optimization Results
	12.4. Homework Problems
	12.5. Lab Assignment #5: Review Chapters 11–12
Appendix: Review of Math with MATLAB
Bibliography
Index