Optimal Control: Basics and Beyond

Title......Page 0
Optimal Control: Basics and Beyond......Page 4
Contents......Page 6
Preface......Page 8
1 CONTROL AS AN OPTIMISATION PROBLEM......Page 12
2 AN EXAMPLE: THE HARVESTING OF A RENEWABLE RESOURCE......Page 13
3 DYNAMIC OPTIMISATION TECHNIQUES......Page 16
4 ORGANISATION OF THE TEXT......Page 18
PART 1: Deterministic Models......Page 20
1 STATE STRUCTURE OPTIMISATION AND DYNAMICPROGRAMMING......Page 22
2 OPTIMAL CONSUMPTION......Page 28
3 PRODUCTION SCHEDULING......Page 31
4 LQ REGULATION......Page 33
5 LQ REGULATION IN THE SCALAR CASE......Page 36
6 DYNAMIC PROGRAMMING IN CONTINUOUS TIME......Page 39
7. OPTIMAL HARVESTING......Page 42
8 LQ REGULATION IN CONTINUOUS TIME......Page 44
9 OPTIMAL TRACKING FOR A DISTURBED LQ MODEL......Page 49
10 OPTIMAL EQUILIBRIUM POINTS: NEIGHBOURING OPTIMALCONTROL......Page 53
1 A FORMALISM FOR THE DYNAMIC PROGRAMMING EQUATION......Page 58
2 INFINITE-HORIZON LIMITS FOR TOTAL COST......Page 60
3 AVERAGE-COST OPTIMALITY......Page 64
4 GROWTH OPTIMALITY......Page 66
5 POLICY IMPROVEMENT......Page 67
6 POLICY IMPROVEMENT AND LQ MODELS......Page 70
7. POLICY IMPROVEMENT FOR THE HARVESTING EXAMPLE......Page 72
1 THE CLASSIC CONTROL FORMULATION......Page 74
2 FILTERS IN DISCRETE TIME: THE SCALAR (SISO) CASE......Page 78
3 FROM DYNAMIC MODEL TO THE INPUT/OUTPUT RELATION:FILTER INVERSION......Page 83
4 FILTERS IN DISCRETE TIME; THE VECTOR (MIMO) CASE......Page 85
5 COMPOSITION AND INVERSION OF FILTERS; z-TRANSFORMS......Page 88
6 FILTERS IN CONTINUOUS TIME......Page 90
7 DYNAMIC MODELS: THE INVERSION OF CONTINUOUS-TIMEFILTERS......Page 91
8 LAPLACE TRANSFORMS......Page 93
9 STABILITY OF CONTINUOUS-TIME FILTERS......Page 94
10 SYSTEMS STABILISED BY FEEDBACK......Page 96
1 STATE-STRUCTURE FOR THE UNCONTROLLED CASE: STABILTY;LINEARISATION......Page 102
2 CONTROL......Page 107
3 THE CAYLEY-HAMILTON THEOREM......Page 110
4 CONTROLLABILITY (DISCRETE TIME)......Page 112
5 CONTROLLABLITY (CONTINUOUS TIME)......Page 115
6 OBSERVABILITY......Page 117
1 INFINITE-HORIZON LIMITS FOR THE LQ REGULATION PROBLEM......Page 122
2. STATIONARY TRACKING RULES......Page 126
3 DIRECT TRAJECTORY OPTIMISATION: WHY THE OPTIMAL FEEDBACK/FEEDFORWARD CONTROL RULE HAS THE FORMIT DOES......Page 127
4 AN ALTERNATIVE FORM FOR THE RICCATI RECURSION......Page 132
5 DIRECT TRAJECTORY OPTIMISATION FOR HIGHER-ORDERCASES......Page 133
6 THE CONCLUSIONS IN TRANSFER FUNCTION TERMS......Page 134
7 DIRECT TRAJECTORY OPTIMISATION FOR THE INPUTOUTPUTFORMULATION: NOTES OF CAUTION......Page 137
CHAPTER 7: The Pontryagin Maximum Principle......Page 142
1 THE PRINCIPLE AS A DIRECT LAGRANGIAN OPTIMISATION......Page 143
2 THE PONTRYAGIN MAXIMUM PRINCIPLE......Page 146
3 TERMINAL CONDITIONS......Page 149
4 MINIMAL TIME AND DISTANCE PROBLEMS......Page 151
5 SOME MISCELLANEOUS PROBLEMS......Page 153
6 THE BUSH PROBLEM......Page 156
7 ROCKET THRUST PROGRAMME OPTIMISATION......Page 158
8 PROBLEMS WHICH ARE PARTIALLY LQ......Page 161
9 CONTROL OF THE INERTIALESS PARTICLE......Page 164
10 CONTROL OF THE INERTIAL PARTICLE......Page 166
11 AVOIDANCE OF THE SfOPPING SEf: A GENERAL RESULT......Page 171
12 CONSTRAINED PATHS: TRANSVERSALITY CONDITIONS......Page 174
13 REGULATION OF A RESERVOIR......Page 176
PART 2: Stochastic Models......Page 178
CHAPTER 8: Stochastic Dynamic Programming......Page 180
1 ONE-STAGE OPTIMISATION......Page 181
2 MULTI-STAGE OPTIMISATION; THE DYNAMIC PROGRAMMINGEQUATION......Page 184
3 STATE STRUCTURE......Page 186
4 THE DYNAMIC PROGRAMMING EQUATION IN CONTINUOUS TIME......Page 188
1 JUMP PROCESSES......Page 190
2 DETERMINISTIC AND PIECEWISE DETERMINISTIC PROCESSES......Page 191
3 THE DERIVATE CHARACTERISTIC FUNCTION......Page 192
4 PROCESSES OF INDEPENDENT INCREMENTS......Page 193
5 WIENER PROCESSES (BROWNIAN MOTION)......Page 196
1 LQG REGULATION WITH PLANT NOISE......Page 200
2 OPTIMAL EXERCISE OF A STOCK OPTION......Page 203
3 A QUEUEING MODEL......Page 204
4 THE HARVESTING EXAMPLE: A BIRTH-DEATH MODEL......Page 205
5 BEHAVIOUR FOR NEAR-DETERMINISTIC MODELS......Page 208
6 THE UNDERSTANDING OF THRESHOLD CHOICE......Page 210
7 CONTINUOUS-TIME LQG PROCESSES AND PASSAGE TO ASTOPPING SET......Page 212
8 THE HARVESTING EXAMPLE: A DIFFUSION MODEL......Page 217
9 THE HARVESTING EXAMPLE: A MODEL WITH A SWITCHINGENVIRONMENT......Page 220
1 THE TRANSFER OF DETERMINISTIC CONCLUSIONS......Page 226
2 AVERAGE-COST OPTIMALITY......Page 228
3 POLICY IMPROVEMENT FOR THE AVERAGE-COST CASE......Page 231
4 MACHINE MAINTENANCE......Page 232
5 CUSTOMER ALLOCATION BETWEEN QUEUES......Page 233
6 ALLOCATION OF SERVER EFFORT BETWEEN QUEUES......Page 234
7 REWARDED SERVICE RATHER THAN PENALISED WAITING......Page 236
8 CALL ROUTING IN LOSS NETWORKS......Page 237
1 AN INDICATION OF CONCLUSIONS......Page 240
2 LQG STRUCTURE AND IMPERFECT OBSERVATION......Page 242
3 THE CERTAINTY EQUIVALENCE PRINCIPLE......Page 245
4 APPLICATIONS: THE EXTENDED CEP......Page 248
5 CALCULATION OF ESTIMATES: THE KALMAN FILTER......Page 250
6 PROJECTION ESTIMATES......Page 253
7 INNOVATIONS......Page 257
9 ESTIMATION AS DUAL TO CONTROL......Page 259
1 STATIONARITY: THE AUTOCOVARIANCE FUNCTION......Page 266
2 THE ACTION OF FILTERS: THE SPECTRAL DENSITY MATRIX......Page 268
3 MOVING-AVERAGE AND AUTOREGRESSIVE REPRESENTATIONS:CANONICAL FACTORISATIONS......Page 272
4 THE WIENER FILTER......Page 275
5 OPTIMISATION CRITERIA IN THE STEADY STATE......Page 276
1 ALLOCATION AND CONTROL......Page 280
2 THE GITTINS INDEX......Page 281
3 OPTIMALITY OF THE GITTINS INDEX POLICY......Page 283
4 EXAMPLES......Page 286
5 RESTLESS BANDITS......Page 288
6 AN EXAMPLE: MACHINE MAINTENANCE......Page 292
7 QUEUEING PROBLEMS......Page 293
CHAPTER 15: Imperfect State Observation......Page 296
1 SUFFICIENCY OF THE POSTERIOR DISTRIBUTION OF STATE......Page 297
2 EXAMPLES: MACHINE SERVICING AND CHANGE-POINTDETECTION......Page 300
PART 3: Risk-Sensitive and H_infinity Criteria......Page 304
1 UTILITY AND RISK-SENSITIVITY......Page 306
2 THE RISK-SENSITIVE CERTAINTY-EQUIVALENCE PRINCIPLE......Page 309
3 SOME SIMPLE EXAMPLES......Page 313
4 THE CASE OF PERFECT STATE OBSERVATION......Page 315
5 THE DISTURBED (NON-HOMOGENEOUS) CASE......Page 318
6 IMPERFECT STATE OBSERVATION: THE SOLUTION OF THEP-RECURSION......Page 319
8 CONTINUOUS TIME......Page 321
9 SOME CONTINUOUS-TIME EXAMPLES......Page 323
10 AVERAGE COST......Page 325
11 TIME-INTEGRAL METHODS FOR THE LEQG MODEL......Page 327
12 WHY DISCOUNT?......Page 328
1 THE Hoo LIMIT......Page 332
2 CHARACTERISTICS OF THE Hoo NORM......Page 335
3 THE Hoo CRITERION AND ROBUSTNESS......Page 337
PART 4: Time-integral Methods and Optimal Stationary Policies......Page 342
1 QUADRATIC INTEGRALS IN DISCRETE TIME......Page 346
2 FACTORISATION AND RECURSION IN THE \'MARKOV\' CASE......Page 348
3 VALUE FUNCTIONS AND RECURSIONS......Page 350
4 A CENTRAL RESULT: THE RICCATI/FACTORISATIONRELATIONSHIP......Page 352
5 POLICY IMPROVEMENT AND SUCCESSIVE APPROXIMATION TOTHE CANONICAL FACTORISATION......Page 353
6 THE INTEGRAL FORMALISM IN CONTINUOUS TIME......Page 355
7 RECURSIONS AND FACTORISATIONS IN CONTINUOUS TIME......Page 357
8 THE PIINR ALGORITHM IN CONTINUOUS TIME......Page 359
1 DERIVATION OF THE SOLUTION FROM THE TIME-INTEGRAL......Page 360
2 EXPRESSION OF THE OPTIMAL CONTROL RULE IN DISCRETETIME......Page 362
3 EXPRESSION OF THE OPTIMAL CONTROL RULE INCONTINUOUS TIME......Page 364
1 THE PROCESS/OBSERVATION MODEL: APPEAL TO CERTAINTY EQUIVALENCE......Page 368
2 PROCESS ESTIMATION IN TIME-INTEGRAL FORM(DISCRETE TIME)......Page 369
3 THE PARALLEL KALMAN FILTER (DISCRETE TIME)......Page 372
4 THE SERIAL KALMAN FILTER (DISCRETE TIME)......Page 374
5 THE CONTINUOUS-TIME CASE; INVALIDITY OF THE SERIAL FILTER......Page 376
6 THE PARALLEL KALMAN FILTER (CONTINUOUS TIME)......Page 377
1 DEDUCTION OF THE TIME-INTEGRAL......Page 382
2 THE STATIONARITY CONDITIONS......Page 384
3 A GENERAL FORMALISM......Page 386
PART 5: Near-determinism and LargeDeviation Theory......Page 390
1 THE LARGE DEVIATION PROPERTY......Page 394
2 THE STATIC CASE: CRAMER\'S THEOREM......Page 395
3 OPERATORS AND SCALING FOR MARKOV PROCESSES......Page 400
4 THE RATE FUNCTION FOR A MARKOV PROCESS......Page 403
5 HAMILTONIAN ASPECTS......Page 406
6 REFINEMENTS OF THE LARGE DEVIATION PRINCIPLE......Page 408
7 EQUILIBRIUM DISTRIBUTIONS AND EXPECTED EXIT TIMES......Page 410
1 SPECIFICATION AND THE RISK-NEUTRAL CASE......Page 416
2 LARGE DEVIATION CONCLUSIONS IN THE RISK-SENSITIVECASE......Page 417
3 AN EXAMPLE: THE INERTIALESS LANDING PROBLEM......Page 419
4 THE OPTIMISATION OF CONSUMPTION OVER A LIFETIME......Page 422
5 A NON-DIFFUSION EXAMPLE: THE REGULATION OF PARTICLENUMBERS......Page 424
1 REDUCTION OF THE CONTROL PROBLEM......Page 426
2 THE LARGE DEVIATION EVALUATION......Page 429
3 THE CASE OF LINEAR DYNAMICS......Page 430
4 LANDING THE INERTIALESS PARTICLE......Page 432
5 OBSTACLE AVOIDANCE FOR mE INERTIALESS PARTICLE......Page 433
6 CRASH AVOIDANCE FOR THE INERTIAL PARTICLE......Page 437
7 THE AVOIDANCE OF ABSORPTION GENERALLY......Page 439
1 SPECIFICATION OF THE PLANT/OBSERVATION PROCESS......Page 442
2 \'CERTAINTY EQUIVALENCE\' AND THE SEPARATION PRINCIPLE......Page 443
3 THE RISK-SENSITIVE MAXIMUM PRINCIPLE......Page 446
5 PURE ESTIMATION: NON-LINEAR FILTERING......Page 448
APPENDIX 1: Notation and Conventions......Page 454
APPENDIX 2: The Structural Basis of Temporal Optimisation......Page 460
APPENDIX 3: Moment Generating Functions; Basic Properties......Page 466
Reforences......Page 468
Index......Page 472
Back Cover......Page 476