Big data analytics: tools and technology for effective planning

Cover......Page 1
Half Title......Page 2
Title Page......Page 4
Copyright Page......Page 5
Table of Contents......Page 6
Preface......Page 8
About the Editors......Page 10
Contributors......Page 12
Chapter 1: Challenges in Big Data......Page 16
  Goals and Challenges of Analyzing Big Data......Page 17
  Paradigm Shifts......Page 18
   k-Means......Page 19
   Economics and Finance......Page 20
   Other Applications......Page 21
  Heterogeneity......Page 22
  Noise Accumulation......Page 23
  Spurious Correlation......Page 24
  Coincidental Endogeneity......Page 26
  Impact on Statistical Thinking......Page 28
  Independence Screening......Page 30
  Dealing with Incidental Endogeneity......Page 31
  Impact on Computing Infrastructure......Page 32
  MapReduce......Page 34
  Impact on Computational Methods......Page 37
  First-Order Methods for Non-Smooth Optimization......Page 38
  Dimension Reduction and Random Projection......Page 39
  Existing Methods......Page 42
   Late Work: Location-Specific Tweet Detection and Topic Summarization in Twitter......Page 44
   RNA Structure and RNA–RNA Association Expectation......Page 45
 References......Page 46
 Key Terminology and Definitions......Page 49
Chapter 2: Challenges in Big Data Analytics......Page 52
 Introduction......Page 53
  Storing the Data......Page 54
  Data Variety......Page 55
  Data Visualization......Page 56
  Talent Management......Page 57
  Company Culture......Page 58
  Why Not a Data Warehouse for Big Data?......Page 59
   Various Data Formats......Page 60
  Issues in Analyzing Big Data......Page 61
   Architecture......Page 62
   MapReduce......Page 63
 Hadoop: Pros and Cons......Page 64
   Apache Avro......Page 65
   Apache Flume......Page 66
   Apache Spark......Page 67
 References......Page 68
Chapter 3: Big Data Reference Model......Page 70
 Introduction into Big Data Management Reference Model......Page 71
  Interaction with Visual Data Views......Page 72
 Introduction to the IVIS4BigData Reference Model......Page 73
  The CRISP4BigData Reference Model......Page 75
   Data Collection, Management, and Curation......Page 76
   Analytics......Page 79
   Data Enrichment......Page 81
   Knowledge Generation and Management......Page 82
   Retention and Archiving......Page 83
  Use Case 2: Emotion Recognition in Video Frames with Hadoop......Page 84
  Hadoop Cluster Installation in the EGI Federated Cloud......Page 85
  MetaMap Annotation Results......Page 86
 References......Page 87
 Key Terminology and Definitions......Page 88
Chapter 4: A Survey of Tools for Big Data Analytics......Page 90
 Survey on Commonly Used Big Data Tools......Page 91
  Group 1: Strong to Moderate Commitment, Strong Potential Growth......Page 93
   Visualization......Page 94
   Unstructured Data......Page 95
  Group 3: Weak Commitment, Good Growth......Page 96
   SQL......Page 97
  Group 4: Strong Commitment, Flat or Declining Growth......Page 98
 Challenges for Big Data Analytics Tools......Page 99
 Tools for Using Big Data......Page 100
   Benefits......Page 101
   Direct Access to Big Data for Analysis......Page 102
  Talend Open Studio......Page 103
  Tableau Desktop and Server......Page 104
  Splice Machine......Page 105
 Conclusions......Page 106
 References......Page 107
Chapter 5: Understanding the Data Science behind Business Analytics......Page 108
 Introduction......Page 109
  Predictive Analytics......Page 110
  Prescriptive Analytics......Page 111
 Analytics Use Case: Customer Churn Prevention......Page 112
  Descriptive Analytics......Page 113
   Techniques Used for Descriptive Analytics......Page 114
  Diagnostic Analytics......Page 117
   Techniques Used for Predictive Analytics......Page 118
   Back-Propagation Weight Adjustment Scheme......Page 121
   Application of Prescriptive Analytics in the Customer Churn Prevention Use Case......Page 122
 Big Data Analytics Architecture......Page 124
  Tools Used for Big Data Analytics......Page 125
   Tableau......Page 126
   R......Page 127
  Prescriptive Analytics......Page 128
 References......Page 129
 Online Sources......Page 130
Chapter 6: Big Data Predictive Modeling and Analytics......Page 132
  Predictive Modeling for Effective Business Planning: A Case Study......Page 133
 Predictive Modeling......Page 135
  Predictive Modeling Process......Page 136
  Selecting and Preparing Data......Page 137
   Feature Vectors......Page 138
  Estimating and Validating the Model......Page 139
 Types of Predictive Models......Page 140
   Types of Regression Algorithms......Page 141
   Use of Decision Trees in Big Data Predictive Analytics......Page 143
   Inference......Page 144
   Use of Random Forests for Big Data......Page 145
  Support Vector Machines......Page 146
  Unsupervised Models: Cluster Analysis......Page 147
   Algorithms for Cluster Analysis......Page 148
   Use of Cluster Analysis for Big Data Predictions......Page 149
  Target Shuffling......Page 152
  Lift Charts......Page 153
  ROC Curves......Page 155
  Bootstrap Sampling......Page 157
  Data Mining Using CRISP-DM Technique......Page 158
  Predictive Analytics Using R Open-Source Tool......Page 159
 Research Trends and Conclusion......Page 162
 References......Page 163
 Introduction......Page 166
 Overview of Deep Learning as a Hierarchical Feature Extractor......Page 167
  Problem Definition......Page 168
  Deep CNNs for Pillar Sequence Design......Page 169
  Action Sequence Learning for Flow Sculpting......Page 171
 Representative Training Data Generation......Page 174
 References......Page 177
Chapter 8: A Framework for Minimizing Data Leakage from Nonproduction Systems......Page 182
  Legal, Business, and Human Factors......Page 183
  Existing Frameworks, Solutions, Products, and Guidelines......Page 185
  Limitations......Page 186
  Freedom of Information: An Organizational View......Page 187
  Questionnaire: Opinion......Page 188
  Six Stages of the Framework, Detailing from Organization to Compliance......Page 197
   Know the Business Data......Page 198
   Data Treatment and Protection......Page 199
   Demonstrate Knowledge......Page 200
  Hypothetical Case Study Scenario......Page 201
   Know the Business Data......Page 202
   Know the Environment......Page 203
   Data Treatment and Protection......Page 204
  Summary of the Impact of the Framework on the Case Study......Page 205
 Glossary......Page 206
 References......Page 207
Chapter 9: Big Data Acquisition, Preparation, and Analysis Using Apache Software Foundation Tools......Page 210
 Introduction......Page 211
 Data Acquisition......Page 213
  Web Scraping......Page 214
  Web Crawling......Page 215
 Data Preprocessing and Cleanup......Page 216
   When to Choose Hadoop over Python or R......Page 217
   Comparison of Hadoop MapReduce, Python, and R......Page 218
  Cleansing Methods and Routines......Page 219
   Loading Data from Flat Files......Page 220
   Merging and Joining Data Sets......Page 221
 Data Analysis......Page 223
   Various Language Support via Apache Hadoop......Page 224
   Apache Spark for Big Data Analytics......Page 225
 Case Study: Dimensionality Reduction Using Apache Spark, an Illustrative Example from Scientometric Big Data......Page 226
  Why SVD? The Solution May Be the Problem......Page 227
   Need for SVD......Page 228
   Relationship between Singular Values and the Input Matrix......Page 229
   SVD Using Hadoop Streaming with Numpy......Page 230
  SVD Using Apache Mahout......Page 231
   Lanczo’s Algorithm......Page 232
  SVD Using Apache Spark......Page 233
   Why Big Data Visualization Is Challenging and Different from Traditional Data Visualization......Page 235
 Conclusions......Page 237
 References......Page 238
  Setting Up Maven’s Memory Usage......Page 239
   build/mvn......Page 240
  Required Cluster Configurations for Spark......Page 241
 Key Terminology and Definitions......Page 242
Chapter 10: Storing and Analyzing Streaming Data: A Big Data Challenge......Page 244
 Streaming Algorithms......Page 245
  Vanilla Model......Page 247
  Sliding Window Model......Page 248
  Cash Register Model......Page 249
  Bloom Filter......Page 250
  Count-Min Sketch......Page 251
  Filtered Space Saving......Page 252
 Suitability of Data Models for Storing Stream Data......Page 253
  Document Stores Model......Page 254
  Industrial Challenges......Page 255
  Open Research Problems......Page 257
 References......Page 258
 Further Readings......Page 260
Chapter 11: Big Data Cluster Analysis: A Study of Existing Techniques and Future Directions......Page 262
 Overview of Cluster Analysis......Page 263
  An Illustrative Example......Page 264
   Termination......Page 265
   Overview......Page 266
  Divisive Clustering......Page 267
   Expected Effects of the Three Vs on the ROCK Clustering Technique......Page 268
   Overview......Page 269
   Expected Effects of the Three Vs on the DBSCAN Clustering Technique......Page 270
  k-Means Clustering Technique......Page 271
   Expected Effects of 3 Vs on the k-Means Clustering Technique......Page 272
   CLIQUE Clustering Technique......Page 273
  Miscellaneous Clustering Techniques......Page 274
   Density-Based Distributed Clustering......Page 275
  Robust Techniques......Page 276
  Clustering Engines......Page 277
 References......Page 278
 Key Terminology and Definitions......Page 279
Chapter 12: Nonlinear Feature Extraction for Big Data Analytics......Page 282
  Algorithm......Page 283
  PCA for Big Data......Page 284
  Problems Associated with Kernel Methods......Page 285
  Kernel Functions as New Features......Page 286
  The Most Common Kernel Functions......Page 287
  Selecting Basis Elements......Page 288
   k-Means......Page 289
   Sparse Greedy Matrix Approximation......Page 290
  Example of Semiparametric SVMs......Page 291
   Stochastic Gradient Descent......Page 292
 Deep Feature Learning......Page 293
   Measuring Discrepancies in AE......Page 294
   Architecture of AEs......Page 295
   Benefits for Big Data......Page 296
  Probabilistic Models......Page 297
   Learning......Page 298
   Contrastive Divergence......Page 299
  Deep Distributed Learning......Page 300
  Updating Parameters in Large-Scale Networks......Page 301
 Ensemble Methods......Page 302
  Trainable Aggregation......Page 304
 References......Page 305
Chapter 13: Enhanced Feature Mining and Classifier Models to Predict Customer Churn for an e-Retailer......Page 308
 Introduction......Page 309
   Source......Page 310
   Store......Page 311
 Existing Models for Customer Churn......Page 312
   Phase 3: Cross-Validation, Business Action, and Performance Tuning......Page 313
   Customer Sales......Page 314
   Frequency......Page 315
  Baseline Methods......Page 316
  Regularization......Page 317
 Cross-Validation......Page 319
  Receiver Operating Characteristics Curve......Page 321
 Conclusions and Future Work......Page 322
 References......Page 323
 Key Terminology and Definitions......Page 324
Chapter 14: Large-Scale Entity Clustering Based on Structural Similarities within Knowledge Graphs......Page 326
 Introduction......Page 327
  Definitions of Big Data......Page 328
   Gartner’s Three Vs......Page 329
  Analytics......Page 330
 Background: Knowledge Graphs......Page 332
  Overview......Page 333
  Freebase Data Model......Page 334
  Data Preprocessing......Page 336
  Formulation of the Similarity Metric......Page 337
  Selection of Clustering Algorithm......Page 338
  Clustering Experiments......Page 340
  Evaluation of Cluster Quality......Page 342
  Cluster Analysis......Page 343
  Future Work: Exploitation of Clusters for Topic Exploration......Page 344
  Related Work......Page 345
 Conclusions......Page 346
 References......Page 347
 Introduction......Page 350
 Iot Paradigm......Page 351
  Main Elements of IoT......Page 352
  Popular Applications of IoT......Page 353
  Security Challenges in the IoT Network......Page 354
  A Model of Systematic Intelligence Formation from Natural Intelligence......Page 355
  Foundations of Human–Computer Interactions......Page 356
  Intelligent Frameworks in IoT......Page 357
  Data Storage and Processing Modules......Page 358
  The Key Devices of Intelligence in IoT......Page 359
  Big Data Analytics Methods for Connected Intelligence with IoT......Page 360
 Evaluation Results......Page 363
 References......Page 365
 Key Terminology and Definitions......Page 367
 Introduction......Page 370
 Big Data-Driven Value Chains......Page 371
  Processing (Data Mining and Fusion)......Page 373
  Consumption (Sharing)......Page 375
 Assembling Value via Heterogeneous Data Fusion and Interoperability......Page 376
  Intermediating Digital Platforms......Page 377
  Linking Big Data with Network Theory......Page 379
 Big Data-Driven Business Planning......Page 381
 Monetization Strategies......Page 383
 Conclusions......Page 384
 References......Page 385
 Introduction......Page 388
  Public Debate Research......Page 389
  Discourse Analysis and Governmentality......Page 391
  A Leveled Approach......Page 393
  Research Demonstration: Public Perception of Amphetamine in Dutch Newspapers......Page 394
 Modularity and the Cross-Media Challenge......Page 401
 Conclusions......Page 403
 References......Page 404
Index......Page 406