Beyond Redundancy: How Geographic Redundancy Can Improve Service Availability and Reliability of Computer-Based Systems

BEYOND REDUNDANCY: How Geographic Redundancy Can Improve Service Availability and Reliability of Computer-Based Systems......Page 5
CONTENTS......Page 9
FIGURES......Page 17
TABLES......Page 21
EQUATIONS......Page 23
PREFACE AND ACKNOWLEDGMENTS......Page 25
ORGANIZATION......Page 26
ACKNOWLEDGMENTS......Page 28
PART 1: BASICS......Page 29
1.1 SERVICE CRITICALITY AND AVAILABILITY EXPECTATIONS......Page 31
1.2 THE EIGHT-INGREDIENT MODEL......Page 32
1.3 CATASTROPHIC FAILURES AND GEOGRAPHIC REDUNDANCY......Page 35
1.4 GEOGRAPHICALLY SEPARATED RECOVERY SITE......Page 39
1.5 MANAGING RISK......Page 40
1.5.2 Risk Treatments......Page 41
1.6 BUSINESS CONTINUITY PLANNING......Page 42
1.7 DISASTER RECOVERY PLANNING......Page 43
1.9 RECOVERY OBJECTIVES......Page 45
1.10 DISASTER RECOVERY STRATEGIES......Page 46
2.1.1 Service Availability......Page 48
2.1.2 Service Reliability......Page 49
2.1.3 Reliability, Availability, and Failures......Page 50
2.2 MEASURING SERVICE AVAILABILITY......Page 53
2.2.1 Total and Partial Outages......Page 54
2.2.2 Minimum Chargeable Disruption Duration......Page 55
2.2.3 Outage Attributability......Page 56
2.2.5 Service Impact and Element Impact Outages......Page 58
2.2.6 Treatment of Planned Events......Page 60
2.3 MEASURING SERVICE RELIABILITY......Page 61
PART 2: MODELING AND ANALYSIS OF REDUNDANCY......Page 63
3.1 TYPES OF REDUNDANCY......Page 65
3.1.1 Simplex Configuration......Page 67
3.1.2 Redundancy......Page 69
3.1.3 Single Point of Failure......Page 71
3.2 MODELING AVAILABILITY OF INTERNAL REDUNDANCY......Page 72
3.2.1 Modeling Active–Active Redundancy......Page 73
3.2.2 Modeling Active Standby Redundancy......Page 77
3.2.3 Service Availability Comparison......Page 79
3.3 EVALUATING HIGH-AVAILABILITY MECHANISMS......Page 80
3.3.2 Recovery Point Objective......Page 82
3.3.4 Capital Expense......Page 83
3.3.6 Discussion......Page 84
4.1 GENERIC EXTERNAL REDUNDANCY MODEL......Page 87
4.1.1 Failure Detection......Page 92
4.1.2 Triggering Recovery Action......Page 94
4.1.3 Traffic Redirection......Page 95
4.1.4 Service Context Preservation......Page 99
4.2 TECHNICAL DISTINCTIONS BETWEEN GEOREDUNDANCY AND CO-LOCATED REDUNDANCY......Page 102
4.3 MANUAL GRACEFUL SWITCHOVER AND SWITCHBACK......Page 103
5.1 REDUNDANCY STRATEGIES......Page 105
5.2 DATA RECOVERY STRATEGIES......Page 107
5.3 EXTERNAL RECOVERY STRATEGIES......Page 108
5.4 MANUALLY CONTROLLED RECOVERY......Page 109
5.5 SYSTEM-DRIVEN RECOVERY......Page 111
5.6 CLIENT-INITIATED RECOVERY......Page 113
5.6.1 Client-Initiated Recovery Overview......Page 114
5.6.2 Failure Detection by Client......Page 116
5.6.3 Client-Initiated Recovery Example: Automatic Teller Machine (ATM)......Page 123
5.6.4 Client-Initiated Recovery Example: A Web Browser Querying a Web Server......Page 124
5.6.5 Client-Initiated Recovery Example: A Pool of DNS Servers......Page 125
6.1 THE SIMPLISTIC ANSWER......Page 126
6.2 FRAMING SERVICE AVAILABILITY OF STANDALONE SYSTEMS......Page 127
6.3.1 Simplifying Assumptions......Page 131
6.3.2 Standalone High-Availability Model......Page 132
6.3.3 Manually Controlled Georedundant Recovery......Page 135
6.3.4 System-Driven Georedundant Recovery......Page 138
6.3.5 Client-Initiated Georedundancy Recovery......Page 139
6.3.6 Complex Georedundant Recovery......Page 141
6.3.7 Comparing the Generic Georedundancy Model to the Simplistic Model......Page 142
6.4 SOLVING THE GENERIC GEOREDUNDANCY MODEL......Page 143
6.4.1 Manually Controlled Georedundant Recovery Model......Page 146
6.4.3 Client-Initiated Georedundant Recovery Model......Page 148
6.5 PRACTICAL MODELING OF GEOREDUNDANCY......Page 149
6.5.1 Practical Modeling of Manually Controlled External System Recovery......Page 150
6.5.2 Practical Modeling of System-Driven Georedundant Recovery......Page 152
6.5.3 Practical Modeling of Client-Initiated Recovery......Page 153
6.6 ESTIMATING AVAILABILITY BENEFIT FOR PLANNED ACTIVITIES......Page 158
6.7 ESTIMATING AVAILABILITY BENEFIT FOR DISASTERS......Page 159
7: UNDERSTANDING RECOVERY TIMING PARAMETERS......Page 161
7.1.1 Understanding and Optimizing TTIMEOUT......Page 162
7.1.2 Understanding and Optimizing TKEEPALIVE......Page 165
7.1.3 Understanding and Optimizing TCLIENT......Page 167
7.1.4 Timer Impact on Service Reliability......Page 168
7.2.1 RTO for Manually Controlled Recovery......Page 169
7.2.2 RTO for System-Driven Recovery......Page 171
7.2.3 RTO for Client-Initiated Recovery......Page 173
7.2.4 Comparing External Redundancy Strategies......Page 174
8.1 OVERVIEW OF DNS......Page 175
8.2 MAPPING DNS ONTO PRACTICAL CLIENT-INITIATED RECOVERY MODEL......Page 176
8.2.1 Modeling Normal Operation......Page 178
8.2.3 Modeling Timeout Failure......Page 179
8.2.4 Modeling Abnormal Server Failure......Page 181
8.3 ESTIMATING INPUT PARAMETERS......Page 182
8.3.1 Server Failure Rate......Page 183
8.3.3 CCLIENT Parameter......Page 186
8.3.5 μCLIENTSFD Parameter......Page 188
8.3.6 μCLIENT Parameter......Page 189
8.3.8 FCLIENT Parameter......Page 190
8.3.9 μGRECOVER and μMIGRATION Parameters......Page 192
8.4 PREDICTED RESULTS......Page 193
8.4.1 Sensitivity Analysis......Page 196
8.5 DISCUSSION OF PREDICTED RESULTS......Page 200
9.1 UNDERSTANDING SOLUTIONS......Page 202
9.1.1 Solution Users......Page 203
9.1.2 Solution Architecture......Page 204
9.2 ESTIMATING SOLUTION AVAILABILITY......Page 205
9.3 CLUSTER VERSUS ELEMENT RECOVERY......Page 207
9.4 ELEMENT FAILURE AND CLUSTER RECOVERY CASE STUDY......Page 210
9.5.3 Traffic Redirection......Page 214
9.6 MODELING CLUSTER RECOVERY......Page 215
9.6.1 Cluster Recovery Modeling Parameters......Page 218
9.6.2 Estimating λSUPERELEMENT......Page 221
9.6.3 Example of Super Element Recovery Modeling......Page 224
PART 3: RECOMMENDATIONS......Page 229
10.1 WHY SUPPORT MULTIPLE SITES?......Page 231
10.2 RECOVERY REALMS......Page 232
10.2.1 Choosing Site Locations......Page 233
10.3 RECOVERY STRATEGIES......Page 234
10.4 LIMP-ALONG ARCHITECTURES......Page 235
10.5.1 Standby Sites......Page 236
10.5.2 N + K Load Sharing......Page 239
10.5.3 Discussion......Page 243
10.6 VIRTUALIZATION, CLOUD COMPUTING, AND STANDBY SITES......Page 244
10.7 RECOMMENDED DESIGN METHODOLOGY......Page 245
11.1 THEORETICALLY OPTIMAL EXTERNAL REDUNDANCY......Page 247
11.2.1 Internal versus External Redundancy......Page 248
11.2.2 Client-Initiated Recovery as Optimal External Recovery Strategy......Page 250
11.2.4 Active–Active Server Operation......Page 251
11.2.5 Optimizing Timeout and Retry Parameters......Page 252
11.2.7 Rapid Context Restoration......Page 253
11.2.10 Network Element versus Cluster-Level Recovery......Page 254
11.3.1 Architecting to Facilitate Planned Maintenance Activities......Page 256
11.3.2 Procedural Considerations......Page 257
12.1 INTERNAL REDUNDANCY REQUIREMENTS......Page 258
12.1.1 Standalone Network Element Redundancy Requirements......Page 259
12.1.2 Basic Solution Redundancy Requirements......Page 260
12.2 EXTERNAL REDUNDANCY REQUIREMENTS......Page 261
12.3.1 Manual Failover Requirements......Page 263
12.3.3 Switchback Requirements......Page 264
12.4.1 System-Driven Recovery......Page 265
12.4.2 Client-Initiated Recovery......Page 267
12.5 OPERATIONAL REQUIREMENTS......Page 270
13.1 GEOREDUNDANCY TESTING STRATEGY......Page 271
13.1.1 Network Element Level Testing......Page 272
13.1.3 Deployment Testing......Page 273
13.2 TEST CASES FOR EXTERNAL REDUNDANCY......Page 274
13.3 VERIFYING GEOREDUNDANCY REQUIREMENTS......Page 275
13.3.2 Test Cases for Manually Controlled Recovery......Page 276
13.3.3 Test Cases for System-Driven Recovery......Page 277
13.3.4 Test Cases for Client-Initiated Recovery......Page 278
13.3.5 Test Cases at the Solution Level......Page 280
13.3.6 Test cases for Operational Testing......Page 281
13.4 SUMMARY......Page 282
14.1 THE HYPOTHETICAL SOLUTION......Page 284
14.1.1 Key Quality Indicators......Page 286
14.2 STANDALONE SOLUTION ANALYSIS......Page 287
14.2.2 Network Element Configuration in Standalone Solution......Page 288
14.2.4 Discussion of Standalone Solution Analysis......Page 289
14.3.1 Identify Factors Constraining Recovery Realm Design......Page 291
14.3.3 Define Recovery Strategies......Page 292
14.3.4 Set Recovery Objectives......Page 293
14.3.5 Architecting Site Redundancy......Page 296
14.5.1 Internal Redundancy Requirements......Page 297
14.5.2 External Redundancy Requirements......Page 300
14.5.3 Manual Failover Requirements......Page 301
14.5.4 Automatic External Recovery Requirements......Page 302
14.6.1 Testing Strategy......Page 305
14.6.2 Standalone Network Element Testing......Page 306
14.6.3 Automatic External Recovery Requirements Testing......Page 307
14.6.4 End-to-End Testing......Page 310
14.6.5 Deployment Testing......Page 311
14.6.6 Operational Testing......Page 312
SUMMARY......Page 313
APPENDIX: MARKOV MODELING OF SERVICE AVAILABILITY......Page 320
ACRONYMS......Page 324
REFERENCES......Page 326
ABOUT THE AUTHORS......Page 328
INDEX......Page 330