Database archiving: how to keep lots of data for a very long time

Cover Page
......Page 1
Copyright Page......Page 2
Dedication Page......Page 3
Preface......Page 4
How This Book Is Organized......Page 5
Who This Book Is For......Page 6
Acknowledgments......Page 8
Database Archiving Overview......Page 10
Selected data objects......Page 11
Can be referenced again if needed......Page 12
Physical documents......Page 13
Electronic files......Page 14
XML documents......Page 15
Emails......Page 16
Databases......Page 17
Physical versus virtual database archiving......Page 18
Operational......Page 19
Reference......Page 20
Discard......Page 21
Types of Data Objects......Page 22
Database Configuration Choices......Page 23
Operational database and archive data store......Page 24
Need to protect the authenticity of data......Page 25
Summary......Page 26
In-House Participants......Page 28
Data Volume Increases......Page 30
Implications of not Keeping Data......Page 32
Data Volume Issues......Page 34
Hitting physical file limits......Page 35
Recovery from data center disasters......Page 36
Change Management......Page 37
Store data in parallel databases......Page 38
Summary......Page 39
Generic Archiving Methodology......Page 40
The Archive Extractor Model......Page 191
Legal requirements......Page 42
Access Auditing......Page 232
Organizational policy......Page 43
Historical significance......Page 44
Scientific significance......Page 45
Trolling for Problems......Page 77
Assessing Metadata Quality......Page 124
Identify Objects to Archive......Page 46
Classification Process......Page 47
Recurring versus nonrecurring archiving......Page 48
Don't move too early or too late......Page 49
Will someone know where to find it?......Page 50
Indefinite retention length......Page 51
What is the start time?......Page 52
Establishing a specific discard policy......Page 53
Determine who Needs Access to Archives and How......Page 54
Determine the form of Archive Objects......Page 55
Preserving context......Page 56
Determining the Consequences of Losing Archive Objects......Page 57
Theft......Page 58
Damage......Page 59
Erosion-related risk factors......Page 60
Loss of meaning......Page 61
Determining Requirements for Making Archive Objects Accessible......Page 62
One place or multiple places......Page 63
Removing objects from the operational environment......Page 64
Discarding objects......Page 65
Requirement changes......Page 66
Summary......Page 67
The Database Archiving Organization......Page 69
Archive Application Data Gathering......Page 70
Archive Data Extraction......Page 71
Archive Access......Page 72
Summary......Page 73
An archiving organization......Page 74
Problems that Lead to Database Archiving Solutions......Page 75
Problems emerging from disasters......Page 76
Scoping the Problem......Page 78
Establishing Basic Requirements......Page 79
Determining that Database Archiving is Part of the Solution......Page 80
A Simple, One-Source Strategy......Page 81
Parallel Applications......Page 82
Retired Applications......Page 83
Data in UNLOAD files......Page 84
Merger and acquisition activities......Page 85
Summary......Page 86
People......Page 88
Authority......Page 90
Repository......Page 91
Metadata management......Page 92
Disk Storage......Page 93
Summary......Page 94
Process Examination......Page 95
Database Topologies......Page 97
Operational replicates......Page 99
Finding Reference Data......Page 100
What to Put into the Archive......Page 101
Documenting Data Sources......Page 102
Summary......Page 103
Metadata Definitions......Page 104
Business Objects......Page 105
Other Helpful Qualifications and Characteristics......Page 106
Tables......Page 108
Reference relationships......Page 109
Where to find Metadata......Page 110
Packaged application metadata repositories......Page 111
Database definitions......Page 112
People......Page 113
Selecting a Version of the Metadata......Page 114
Getting the right data......Page 115
Sensitive data classification......Page 116
Data quality......Page 117
Documenting Metadata......Page 118
Keeping up with Changes......Page 119
Summary......Page 120
Matching Data to Metadata......Page 121
Relational data......Page 122
Assessing Data Quality......Page 123
Documenting Validation Activities......Page 125
Summary......Page 126
Independence from Application Programs......Page 127
Transforming to an Independent Form......Page 128
Metadata......Page 129
Multiple Data Sources......Page 130
Independence from Data Formats......Page 131
Summary......Page 133
The Source Data Model......Page 134
Object relationships......Page 136
Relationship links......Page 137
Order of definition......Page 138
The Target Data Model......Page 139
Business Objects in the Archive......Page 140
Normalization......Page 141
Reusing identifiers......Page 142
Duplicate keys......Page 143
Model Representations......Page 144
Summary......Page 145
Extract Policies......Page 146
Format of selection criteria......Page 147
Data Disposition on Extraction......Page 148
Leave the data with no changes......Page 149
Continuous archiving......Page 150
Archive Storage Policies......Page 151
Device types......Page 152
Protection from Wrongful Access......Page 153
Discard Criteria......Page 154
Disposition of Data......Page 155
Summary......Page 156
Archiving System Strategies for Handling Metadata Changes......Page 157
Multiple Data Definition Strategy......Page 158
Minor Metadata Changes......Page 159
No-impact changes......Page 160
Adding and dropping columns......Page 161
Semantic changes......Page 162
Adding a table......Page 163
Combining or splitting tables......Page 164
Upgrades to Packaged Applications......Page 165
Changes to the Archive Transformed View......Page 166
Archive strategy for chnages......Page 167
Changes to Extract Policies......Page 168
Policy for frequency of extract execution......Page 169
Changing devices......Page 170
Discard policy......Page 171
Maintaining an Audit Trail of Changes......Page 172
Summary......Page 173
The Archive Data Store......Page 174
DBMSs of different types......Page 175
Custom archive DBMS......Page 176
Use an external repository......Page 177
Independence from Operational Systems......Page 178
Ability to Store Enriched Metadata......Page 179
Ability to Store Data in Units with Metadata Affinity......Page 180
e-Discovery Readiness Test......Page 237
Ability to Add Data Without Disrupting Previously Stored Objects......Page 181
Search Capabilities......Page 182
Access from Application Programs......Page 183
Protect from Unauthorized Updates or Deletes......Page 184
Selective delete for Discard......Page 185
Transparent Access to Data from Application Programs......Page 186
High Performance on Data Access......Page 187
Summary......Page 188
The Archive Data Extraction Component......Page 190
Double-Process Model......Page 192
Form of the Program......Page 193
Online or Batch......Page 194
Execution Options......Page 196
Limits on time run......Page 197
Locks and Commits......Page 198
Restarting Versus Starting Over......Page 200
Workspace Management......Page 201
Other Considerations......Page 202
Capacity Planning......Page 203
Post-Extract Functions......Page 205
Summary......Page 206
Unit of Discard: The Business Object......Page 207
Scope of Discard......Page 208
DBMS Data Store......Page 209
File Data Store......Page 210
Implementation form......Page 211
Interruptions......Page 212
Forensic Delete......Page 213
Execution Options......Page 214
Performance......Page 215
Capacity Issues......Page 216
Execution Log......Page 217
The audit......Page 218
Summary......Page 219
Direct Programming Access......Page 220
Data storage issues......Page 221
Access Through Report Generators and BI Tools......Page 222
Selective Data Unload......Page 223
Data formats......Page 224
Accessing Original Data......Page 225
Returning data to its original data source......Page 226
Access Authorities......Page 227
Performance......Page 228
Summary......Page 229
Ongoing Auditing and Testing......Page 231
Access audit......Page 233
Extract Auditing......Page 234
Policy Auditing......Page 235
Continuous regression testing......Page 236
Summary......Page 238
DBMS archive data store......Page 239
File archive-level data store......Page 240
Managing the Archive Repository......Page 241
Managing the repository data......Page 242
Using Hosted Solutions......Page 243
User Management Functions......Page 244
Different Group of Users......Page 245
Permissions for Using the Archive Data Store......Page 246
Summary......Page 247
Retired Applications......Page 249
Special challenges......Page 250
One is archiving, one is not archiving......Page 251
Required Qualifications......Page 252
Use of the lawsuit application archive......Page 253
Business Intelligence Data......Page 254
Logs, Audit Trails, and Other Miscellaneous Stuff......Page 255
Summary......Page 256
Final Thoughts......Page 257
Generic Archiving Checklist......Page 258
Goals of a Database Archiving System......Page 261
Job Responsibilities......Page 262
Glossary......Page 264
References......Page 267
A......Page 269
D......Page 270
O......Page 274
Y......Page 275