Artificial Intelligence for Big Data: Complete guide to automating Big Data solutions using Artificial Intelligence techniques

Cover......Page 1
Copyright and Credits......Page 3
Packt Upsell......Page 5
Contributors......Page 6
Table of Contents......Page 8
Preface......Page 15
Chapter 1: Big Data and Artificial Intelligence Systems......Page 22
Results pyramid......Page 23
Storage......Page 24
Speed information storage......Page 25
Best of both worlds......Page 26
Big Data......Page 27
Evolution from dumb to intelligent machines......Page 29
Types of intelligence......Page 30
Big data frameworks......Page 31
Batch processing......Page 32
Real-time processing......Page 33
Frequently asked questions......Page 34
Summary......Page 36
Chapter 2: Ontology for Big Data......Page 37
Human brain and Ontology......Page 38
Ontology of information science......Page 40
Ontology properties......Page 41
Advantages of Ontologies......Page 42
Components of Ontologies......Page 43
The role Ontology plays in Big Data......Page 44
Goals of Ontology in big data......Page 46
RDF—the universal data format......Page 47
RDF containers......Page 50
RDF properties......Page 51
Using OWL, the Web Ontology Language......Page 52
SPARQL query language......Page 54
Generic structure of an SPARQL query......Page 56
Additional SPARQL features......Page 57
Building intelligent machines with Ontologies......Page 58
Ontology learning......Page 61
Ontology learning process......Page 62
Frequently asked questions......Page 64
Summary......Page 65
Chapter 3: Learning from Big Data......Page 66
Supervised and unsupervised machine learning......Page 67
The Spark programming model......Page 72
The transformer function......Page 75
Pipeline......Page 76
Regression analysis......Page 77
Least square method......Page 78
Logistic regression classification technique......Page 82
Polynomial regression......Page 84
Backward elimination......Page 86
Data clustering......Page 87
The K-means algorithm......Page 89
K-means implementation with Spark ML......Page 91
Data dimensionality reduction......Page 92
Matrix theory and linear algebra overview......Page 94
SVD with Spark ML......Page 98
The principal component analysis method......Page 100
Implementing SVD with Spark ML......Page 101
Content-based recommendation systems......Page 102
Frequently asked questions......Page 107
Summary......Page 108
Chapter 4: Neural Network for Big Data......Page 109
Fundamentals of neural networks and artificial neural networks......Page 110
Perceptron and linear models......Page 112
Component notations of the neural network......Page 113
Mathematical representation of the simple perceptron model......Page 114
Activation functions......Page 116
Sigmoid function......Page 117
ReLu......Page 118
Feed-forward neural networks......Page 120
Gradient descent and backpropagation......Page 122
Gradient descent pseudocode......Page 126
Backpropagation model ......Page 127
Overfitting......Page 129
The need for RNNs......Page 131
Training an RNN......Page 132
Frequently asked questions......Page 134
Summary......Page 136
Chapter 5: Deep Big Data Analytics......Page 137
Deep learning basics and the building blocks......Page 138
Gradient-based learning......Page 140
Backpropagation......Page 142
Non-linearities......Page 144
Dropout......Page 146
Building data preparation pipelines......Page 147
Practical approach to implementing neural net architectures......Page 154
Hyperparameter tuning......Page 157
Learning rate......Page 158
Number of training iterations......Page 159
Number of epochs......Page 160
Experimenting with hyperparameters with Deeplearning4j......Page 161
Distributed computing......Page 166
Distributed deep learning......Page 168
API overview......Page 169
TensorFlow......Page 171
Keras......Page 172
Frequently asked questions......Page 173
Summary......Page 175
Chapter 6: Natural Language Processing......Page 176
Natural language processing basics......Page 177
Removing stop words......Page 179
Porter stemming......Page 181
Lancaster stemming......Page 182
Dawson stemming......Page 183
N-grams......Page 184
One hot encoding......Page 185
TF-IDF......Page 186
CountVectorizer......Page 189
CBOW......Page 190
Skip-Gram model......Page 192
Applying NLP techniques......Page 193
Text classification......Page 194
Introduction to Naive Bayes' algorithm......Page 195
Random Forest......Page 196
Naive Bayes' text classification code example......Page 197
Implementing sentiment analysis......Page 199
Frequently asked questions......Page 201
Summary......Page 202
Chapter 7: Fuzzy Systems......Page 203
Fuzzy logic fundamentals......Page 204
Fuzzy sets and membership functions......Page 205
Attributes and notations of crisp sets......Page 206
Operations on crisp sets......Page 207
Fuzzification......Page 208
Fuzzy inference ......Page 211
Adaptive network......Page 212
ANFIS architecture and hybrid learning algorithm......Page 213
Fuzzy C-means clustering......Page 216
NEFCLASS......Page 220
Frequently asked questions......Page 222
Summary......Page 223
Chapter 8: Genetic Programming......Page 224
Genetic algorithms structure......Page 227
KEEL framework......Page 230
Encog API structure......Page 235
Introduction to the Weka framework......Page 239
Preprocess......Page 244
Classify......Page 247
Attribute search with genetic algorithms in Weka......Page 252
Summary......Page 255
Chapter 9: Swarm Intelligence......Page 256
Swarm intelligence ......Page 257
Self-organization......Page 258
Division of labor......Page 260
Advantages of collective intelligent systems......Page 261
Design principles for developing SI systems......Page 262
The particle swarm optimization model......Page 263
PSO implementation considerations ......Page 266
Ant colony optimization model......Page 267
MASON Library......Page 270
MASON Layered Architecture......Page 271
Opt4J library......Page 275
Applications in big data analytics......Page 277
Multi-objective optimization......Page 280
Frequently asked questions......Page 281
Summary......Page 282
Chapter 10: Reinforcement Learning......Page 283
Reinforcement learning algorithms concept......Page 284
Markov decision processes......Page 288
Dynamic programming and reinforcement learning......Page 290
Learning in a deterministic environment with policy iteration......Page 291
Q-Learning......Page 294
SARSA learning......Page 303
Deep reinforcement learning......Page 305
Frequently asked questions......Page 306
Summary......Page 307
Chapter 11: Cyber Security......Page 308
Big Data for critical infrastructure protection......Page 309
Data collection and analysis......Page 310
Anomaly detection ......Page 311
Corrective and preventive actions ......Page 312
Conceptual Data Flow......Page 313
Hadoop Distributed File System......Page 314
MapReduce......Page 315
Hive......Page 316
Understanding stream processing......Page 317
Stream processing semantics......Page 318
Spark Streaming......Page 319
Kafka......Page 320
Lateral movement......Page 323
AI-based defense ......Page 324
Understanding SIEM......Page 326
Visualization attributes and features......Page 328
Splunk......Page 329
Splunk Light......Page 330
Frequently asked questions......Page 333
Summary......Page 335
Chapter 12: Cognitive Computing......Page 336
Cognitive science......Page 337
Cognitive Systems......Page 341
A brief history of Cognitive Systems......Page 342
Goals of Cognitive Systems......Page 344
Cognitive Systems enablers......Page 346
Application in Big Data analytics......Page 347
Cognitive intelligence as a service......Page 349
IBM cognitive toolkit based on Watson......Page 350
Watson-based cognitive apps......Page 351
Setting up the prerequisites......Page 354
Developing a language translator application in Java......Page 356
Frequently asked questions......Page 359
Summary......Page 360
Other Books You May Enjoy......Page 362
Index......Page 365