The Practitioner's Guide to Data Quality Improvement

Front Matter\r......Page 1
Copyright......Page 4
Foreword......Page 5
Data Quality Knowledge Transfer......Page 7
Why You Should Still Buy \"Enterprise Knowledge Management - The Data Quality Approach\"......Page 8
Putting My Money Where My Mouth Is......Page 9
Overview of the Book......Page 11
Contact Me......Page 12
Acknowledgments......Page 14
About the Author......Page 16
Business Impacts of Poor Data Quality......Page 17
Information Value and Data Quality Improvement......Page 19
Business Expectations and Data Quality......Page 20
Qualifying Impacts......Page 21
Credit Risk......Page 23
Underbilling and Revenue Assurance......Page 24
Integration Risks......Page 25
Fraud Risks......Page 26
Financial......Page 27
Productivity......Page 28
Business Impact Analysis......Page 29
Additional Impact Categories......Page 30
Impact Taxonomies and Iterative Refinement......Page 31
Summary: Translating Impact into Performance......Page 32
The Virtuous Cycle of Data Quality......Page 33
Business Impact Analysis......Page 35
Data Quality Requirements Analysis......Page 36
Engaging the Business Data Consumer......Page 37
Metadata Management......Page 38
Inspection and Monitoring......Page 39
Data Quality Incident and Performance Reporting......Page 40
Root Cause Analysis......Page 41
Identity Resolution......Page 42
Business Clients......Page 43
Data Stewards and Data Quality Analysts......Page 44
Operations Staff......Page 45
Parsing......Page 46
Identity Resolution......Page 47
Data Cleansing......Page 48
Inspection and Monitoring......Page 49
Summary......Page 50
The Data Quality Strategy......Page 51
Data Quality Expectations......Page 54
Policies......Page 55
Standards......Page 56
Performance Management......Page 57
Initial......Page 58
Defined......Page 59
Data Quality Expectations......Page 60
Dimensions of Data Quality......Page 61
Procedures......Page 62
Standards......Page 63
Performance Management......Page 64
Summary......Page 65
Planning Initiatives......Page 68
Performance Management and Key Performance Indicators......Page 69
Process Improvement......Page 70
rganizational Change......Page 71
Strategic Planning......Page 72
Centralized versus Decentralized Control......Page 74
Enterprise Architecture......Page 75
Build versus Buy......Page 76
Operational and Application Initiatives......Page 77
Business Intelligence and Data Warehousing......Page 78
Scoping Issues......Page 79
Vertical versus Horizontal......Page 80
Summary......Page 81
Developing A Business Case and A Data Quality Road Map......Page 82
Return on the Data Quality Investment......Page 83
Finding the Business Impacts......Page 84
Clarification of Business Objectives......Page 85
Identifying Data Flaws......Page 86
Researching Costs......Page 87
Correlating Impacts and Causes......Page 88
The Impact Matrix......Page 89
Mapping Impacts to Data Flaws......Page 90
Estimating the Value Gap......Page 91
Prioritizing Actions......Page 94
The Data Quality Road Map......Page 96
Operationalizing Data Quality Management......Page 97
Assess, Tune, Optimize......Page 98
Accountability, Responsibility, and Management......Page 99
Articulating Accountability: The RACI Matrix......Page 100
The Life Cycle of the Data Quality Program......Page 101
Summary......Page 105
Metrics and Performance Improvement......Page 106
Performance-Oriented Data Quality......Page 107
Developing Data Quality Metrics......Page 108
Business Relevance......Page 109
Drill-Down Capability......Page 110
Measurement and Key Data Quality Performance Indicators......Page 111
Key Data Quality Performance Indicators......Page 112
Hierarchical Drill-Through......Page 113
Variation and Control......Page 114
Control Charts......Page 116
The Pareto Principle......Page 117
Building a Control Chart......Page 119
Percentage Nonconforming......Page 120
Number of Nonconformities per Item......Page 121
Example: Invalid Records......Page 122
The Goal of Statistical Process Control......Page 124
Zone Tests......Page 125
Finding Special Causes......Page 126
Summary......Page 127
Data Governance......Page 129
The Data Quality Charter......Page 130
Mission and Guiding Principles......Page 131
Roles and Responsibilities......Page 132
Data Quality Oversight Board......Page 133
Data Coordination Council......Page 134
Data Standards/Metadata Advisory Group......Page 135
Data Stewardship......Page 136
Data Coordination Council Certification Endorsement......Page 139
Ongoing Validation......Page 140
Data Governance and Federated Communities......Page 141
Summary......Page 142
Dimensions of Data Quality......Page 143
What Are Dimensions of Data Quality?......Page 144
Intrinsic Dimensions......Page 145
Classifying Dimensions......Page 146
Describing Data Quality Dimensions......Page 148
Lineage......Page 149
Semantic Consistency......Page 151
Completeness......Page 152
Consistency......Page 153
Currency......Page 154
Reasonableness......Page 155
Qualitative Dimensions......Page 156
Summary......Page 160
Data Requirements Analysis......Page 161
Business Uses of Information and Business Analytics......Page 162
Business Uses of Information......Page 163
Business Analytics......Page 164
Business Drivers and Data Dependencies......Page 165
What Is Data Requirements Analysis?......Page 166
Availability......Page 167
Identifying the Business Contexts......Page 168
Conduct Stakeholder Interviews......Page 170
Synthesize Requirements......Page 171
Source-to-Target Mapping......Page 173
Data Quality Assessment......Page 174
Data Quality Dimensions - Quick Review......Page 175
Data Element Rules......Page 176
Table and Cross-Table Rules......Page 177
Summary......Page 178
Metadata and Data Standards......Page 180
Challenges......Page 181
Data Standards......Page 182
Data Standards Process......Page 183
Metadata Management......Page 184
Data Concepts......Page 186
Definitions......Page 187
Semantics......Page 188
Value Domains......Page 189
Mappings......Page 191
Data Elements......Page 192
Critical Data Elements......Page 193
Data Formats......Page 194
Aliases/Synonyms......Page 195
Business Metadata......Page 196
Business Rules......Page 197
Data Harmonization......Page 198
Summary......Page 202
Data Quality Assessment......Page 203
Select a Business Process and Assess Scope......Page 204
Preliminary Preparation......Page 205
Interview Business Data Consumers......Page 206
Review Collected Documents and Metadata......Page 207
Identify Dimensions of Data Quality......Page 208
Critical Data Elements......Page 209
Preparing the Data Analysis Environment......Page 210
Column Analysis......Page 211
Table Analysis......Page 212
Documenting Potential Anomalies......Page 213
List and Review......Page 214
Prepare Assessment Report......Page 215
Identify Remediation Tasks......Page 217
Summary Rapid Data Assessment - Tangible Results......Page 218
Remediation and Improvement Planning......Page 219
Triage......Page 220
The Prioritization Matrix......Page 221
Gathering Knowledge......Page 222
Assigning Criticality......Page 223
The Information Flow Map......Page 224
The IP-MAP Described......Page 225
Constructing an IP-MAP......Page 226
Identifying the Source of Errors......Page 227
Remediation......Page 228
Root Cause Elimination......Page 229
Summary......Page 230
Data Quality Service Level Agreements......Page 231
Business Drivers and Success Criteria......Page 232
Business Rules and Data Dependencies......Page 233
Defining Data Quality Acceptability Levels......Page 234
Identifying Data Quality Rules......Page 235
Successive Rule Refinement......Page 236
The Challenge of Semantics in Free-Form Text......Page 237
Establishing Data Quality Control......Page 239
Data Handoffs......Page 240
Defining the Data Quality Service Level Agreement......Page 241
Inspection and Monitoring......Page 242
Measurement Processes......Page 243
Data Quality Incident Reporting and Tracking......Page 244
Automating the Collection of Metrics......Page 246
Reporting the Scorecard......Page 247
The Data Quality Issues View......Page 249
Managing Scorecard Views......Page 250
Summary - Managing Using the Data Quality Scorecard......Page 251
Data Profiling......Page 253
Application Contexts for Data Profiling......Page 254
Anomaly Analysis......Page 255
Metadata Compliance and Data Model Integrity......Page 256
Column Analysis......Page 257
Cross-Column Analysis......Page 258
Data Rule Validation......Page 259
Domain Discovery and Analysis......Page 260
Analyzing Anomalies......Page 261
Cross-Column Anomalies......Page 262
Data Quality Rule Discovery......Page 263
Column-Oriented Rules......Page 264
Cross-Column Rule Discovery......Page 265
Metadata Compliance and Data Model Integrity......Page 266
Data Model Integrity......Page 267
Data Steward......Page 268
Selecting a Data Set for Analysis......Page 269
The Question of Sampling......Page 270
Summary......Page 271
Parsing and Standardization......Page 272
Data Error Paradigms......Page 273
The Role of Metadata......Page 275
Mapping ``Data´´ to ``Information´´......Page 277
What Is a Token?......Page 278
Types of Tools......Page 279
Error Tokens......Page 280
Standardized Formats......Page 281
Error Correction......Page 282
Defining Rules......Page 283
Transformations......Page 284
Automation and Knowledge Bases......Page 285
The Proactive versus Reactive Paradox......Page 286
Data Supplier Management......Page 287
Summary......Page 288
Entity Identity Resolution......Page 290
The Lure of Data Correction......Page 291
Root Causes of Duplicates......Page 292
What Is an Entity?......Page 293
Identifying Attributes......Page 294
Characteristics of Identifying Attributes......Page 295
Similarity Analysis and the Matching Process......Page 296
Parsing and Standardization......Page 297
Edit Distance......Page 298
Phonetic Comparison......Page 299
False Positives, False Negatives, and Thresholding......Page 300
Scoring Precision......Page 301
Survivorship......Page 302
Quality of the Values......Page 303
Entity Search and Match and Computational Complexity......Page 304
Duplicate Reduction and Elimination......Page 305
Householding......Page 306
Summary......Page 307
Inspection, Monitoring, Auditing, and Tracking......Page 309
Instituting Inspection and Monitoring: Technology and Process......Page 310
Defining Controls......Page 311
Data Quality Inspection Points......Page 313
Data Quality Business Rules......Page 314
Inspection Services......Page 317
Monitoring......Page 318
Incident Reporting, Notifications, and Issue Management......Page 319
Notification and Escalation......Page 320
Tracking......Page 321
Putting It Together......Page 322
Data Enhancement......Page 323
The Value of Enhancement......Page 324
Inlined Enhancement......Page 325
Temporal Data......Page 326
Context Data......Page 327
Psychographic Data......Page 328
Enhancement through Standardization......Page 329
Enhancement through Context......Page 330
Enhancement through Data Merging......Page 331
Examples......Page 332
Summary: Qualifying Data Sources for Enhancement......Page 334
Master Data Management and Data Quality......Page 336
What Is Master Data?......Page 337
What Is Master Data Management?......Page 339
\"Golden Record\" or \"Unified View\"?......Page 340
Master Data Management as a Tool......Page 341
MDM: A High-Level Component Approach......Page 342
The Master Data Repository......Page 343
Master Data Services......Page 344
Reference Information Management......Page 345
Operational Usage......Page 346
Analytic Usage......Page 347
Master Data Management Architectures......Page 348
Registry......Page 349
Full Repository......Page 350
Hybrid Approach......Page 351
Identifying Master Data......Page 352
Master Data Services......Page 353
Data Access Services......Page 354
Cleansing and Consolidation Services......Page 356
Administration and Stewardship Services......Page 357
Summary: Approaching MDM and Data Quality......Page 358
Developing the Data Quality Business Case......Page 360
Aspect of the Data Quality Program......Page 362
The Data Quality Scan: Assessing Organizational Maturity......Page 365
Collaboration with Enterprise Initiatives......Page 367
Developing the Data Quality Road Map......Page 368
Defining Metrics......Page 369
Instituting Data Governance......Page 371
Defining Dimensions of Data Quality......Page 373
Data Requirements Analysis......Page 374
Metadata Management......Page 375
Data Quality Assessment......Page 377
Remediation and Improvement Processes......Page 378
The Data Quality Service Level Agreement......Page 380
Data Profiling......Page 382
Parsing and Standardization......Page 384
Identity Resolution......Page 385
Auditing, Monitoring, and Tracking......Page 387
Data Enhancement......Page 389
Master Data Management......Page 390
Summary......Page 392
B......Page 393
C......Page 394
D......Page 395
G......Page 398
I......Page 399
M......Page 400
P......Page 402
R......Page 403
S......Page 404
V......Page 405
Z......Page 406