Machine Learning, Animated (Chapman & Hall/CRC Machine Learning & Pattern Recognition)

CHAPTER 1 ◾ Installing Anaconda and Jupyter Notebook

        1.1 Why Python for Machine Learning?

            1.1.1 The Rise of Python

            1.1.2 Python for Machine Learning

        1.2 Installing Anaconda

            1.2.1 Installing Anaconda in Windows

            1.2.2 Installing Anaconda in macOS

            1.2.3 Installing Anaconda in Linux

            1.2.4 Difference between Conda-install and Pip-install

        1.3 Virtual Environment for this Book

            1.3.1 Create the Virtual Environment MLA

            1.3.2 Activate the Virtual Environment

            1.3.3 De-activate the Virtual Environment

        1.4 Set Up Jupyter Notebook in the Virtual Environment

            1.4.1 Write Python in Jupyter Notebook

            1.4.2 Issue Commands in Jupyter Notebook

        1.5 File System for the Book

        1.6 Glossary

        1.7 Exercises

    CHAPTER 2 ◾ Creating Animations

        2.1 Create Plots with Matplotlib

            2.1.1 A Single Line Plot

            2.1.2 Multiple Lines in the Same Plot

        2.2 Create Subplots

            2.2.1 Create Individual Plots

            2.2.2 Create Subplots

        2.3 Create Animated Plots

            2.3.1 Generate Annual Line Plots

            2.3.2 Animate the Plots

        2.4 Create Animated Bar Charts

            2.4.1 Create a Horizontal Bar Chart

            2.4.2 Generate Annual Bar Charts

            2.4.3 Animate the Bar Charts

        2.5 Put Bar Charts and Plots Side by Side

            2.5.1 Combine a Bar Chart and a Plot

            2.5.2 Create an Animation of the Combined Pictures

        2.6 Animated Pie Charts

            2.6.1 Create a Pie Chart

            2.6.2 Generate Annual Pie Charts

            2.6.3 Animate the Combined Pie Charts and Plots

        2.7 Appendix: Download and Clean Up the GDP Data

        2.8 Glossary

        2.9 Exercises

SECTION II Machine Learning Basics

    CHAPTER 3 ◾ Machine Learning: An Overview

        3.1 ML: A New Paradigm for AI

            3.1.1 What is AI?

            3.1.2 Rule-Based AI

            3.1.3 What is ML? Why ML Matters?

            3.1.4 Why is ML so Popular?

        3.2 Different Types of ML

            3.2.1 Supervised Learning

            3.2.2 Unsupervised Learning

            3.2.3 Reinforcement Learning

        3.3 Deep Reinforcement Learning

            3.3.1 Deep Learning

            3.3.2 Combine Deep Learning and Reinforcement Learning

        3.4 Apply ML in the Real World

            3.4.1 The Delivery Route Problem

            3.4.2 Try the Problem before You Know the Answer

        3.5 Glossary

        3.6 Exercises

    CHAPTER 4 ◾ Gradient Descent – Where Magic Happens

        4.1 Optimization through Grid Search

            4.1.1 How Grid Search Achieves Optimization

            4.1.2 Curse of Dimensionality and Directional Grid Search

        4.2 Gradient Descent

        4.3 Use Tensorflow to Calculate Gradients

            4.3.1 Install TensorFlow

            4.3.2 Calculate Gradients Using TensorFlow

            4.3.3 Gradient Descent Optimization with TensorFlow

            4.3.4 Animate the Optimization Process

        4.4 Choose the Right Learning Rate

            4.4.1 When the Learning Rate is Too Large

            4.4.2 When the Learning Rate is Too Small

        4.5 Compare Learning Rates

            4.5.1 Combine Animations

            4.5.2 Subplots of Different Stages

        4.6 Glossary

        4.7 Exercises

    CHAPTER 5 ◾ Introduction to Neural Networks

        5.1 Anatomy of A Neural Network

            5.1.1 Elements of a Neural Network

            5.1.2 How Does a Neural Network Learn?

            5.1.3 Make Predictions

        5.2 Animate the Learning Process

            5.2.1 Generate Graphs

            5.2.2 Create Animation Based on Graphs

            5.2.3 Subplots of Different Stages

        5.3 Create A Neural Network with Keras

            5.3.1 Construct the Model

            5.3.2 Compile and Train the Model

            5.3.3 Make Predictions

        5.4 Customize Training with GradientTape

            5.4.1 Construct the Model

            5.4.2 Train the Model

            5.4.3 Make Predictions

        5.5 Glossary

        5.6 Exercises

    CHAPTER 6 ◾ Activation Functions

        6.1 Why Do We Need Activation Functions?

            6.1.1 Construct a Neural Network

            6.1.2 Learn a Nonlinear Relation without Activation

        6.2 The ReLU Activation Function

            6.2.1 What is ReLU?

            6.2.2 Animate the ReLU Function

            6.2.3 Use ReLU to Model Nonlinearity

        6.3 The Sigmoid Activation Function

            6.3.1 Plot the Sigmoid Function

            6.3.2 Animate the Sigmoid Function

            6.3.3 Combine Animations

            6.3.4 A Picture with Subplots of Different Stages

        6.4 The Softmax Activation Function

            6.4.1 What is the Softmax Function?

            6.4.2 A Diagram of the Softmax Function

        6.5 Glossary

        6.6 Exercises

SECTION III Binary and Multi-Category Classifications

    CHAPTER 7 ◾ Binary Classifications

        7.1 What Is A Binary Classification Problem

            7.1.1 Sigmoid Activation in Binary Classifications

            7.1.2 The Binary Cross-Entropy Loss Function

        7.2 Process Image Data

            7.2.1 Download Data

            7.2.2 Convert NumPy Arrays to Pictures and Back

            7.2.3 Match Pictures with Labels

        7.3 Binary Classification with A Logit Regression

            7.3.1 Prepare the Data

            7.3.2 Train the Logit Model

            7.3.3 Predict Using the Logit Model

        7.4 Binary Classification with A Simple Neural Network

            7.4.1 Train and Test Using a Neural Network

            7.4.2 Focus on Two Examples

            7.4.3 Diagrams of the Network and Predictions

            7.4.4 Animate the Training Process

            7.4.5 Animate the Predictions for the Deer

        7.5 Combine the Animations

            7.5.1 Animate the Two Predictions

            7.5.2 Subplots

        7.6 Binary Classification with A Deep Neural Network

        7.7 Appendix: Load CIFAR10 from Tensorflow Directly

        7.8 Glossary

        7.9 Exercises

    CHAPTER 8 ◾ Convolutional Neural Networks

        8.1 What Are Convolutional Neural Networks (CNNs)?

            8.1.1 Our Running Example

            8.1.2 A Horizontal Filter

        8.2 Convolution Operations

            8.2.1 Calculations in a Convolution Operation

            8.2.2 Animate the Convolution Operations

            8.2.3 Subplots

        8.3 Stride and Padding

            8.3.1 A Filter without Padding and a Stride of 2

            8.3.2 Animate the Diagonal Filter

            8.3.3 Animate the Diagonal Filter Convolution Operation

            8.3.4 Subplots for Strides

        8.4 Combine the Two Animations

            8.4.1 Combine the Animations

        8.5 Max Pooling

        8.6 Binary Classifications with Convolutional Layers

        8.7 Glossary

        8.8 Exercises

    CHAPTER 9 ◾ Multi-Category Image Classifications

        9.1 Image Augmentations

            9.1.1 The Keras Image Generator

            9.1.2 Visualize Image Augmentations

        9.2 What Is Multi-Category Classification?

            9.2.1 One-Hot Encoder for Labels

            9.2.2 The Activation and Loss Functions

        9.3 Train the Multi-Category Classification Model

            9.3.1 Load the Full Data Set

            9.3.2 Convert Labels to One-Hot Variables

            9.3.3 Train the Model

            9.3.4 Evaluate the Model

        9.4 Animate the Learning Process

            9.4.1 Select Example Pictures

            9.4.2 Animate Prediction Changes

            9.4.3 Subplots of the Predictions on the Truck Image

            9.4.4 Animate Predictions on the Frog Image

            9.4.5 Subplots of the Predictions on the Frog Image

            9.4.6 Combine the Animations

        9.5 Glossary

        9.6 Exercises

SECTION IV Developing Deep Learning Game Strategies

    CHAPTER 10 ◾ Deep Learning Game Strategies

        10.1 Get Started with the OpenAI Gym Environment

            10.1.1 Basic Elements of a Game Environment

            10.1.2 The Frozen Lake Game

            10.1.3 Play the Frozen Lake Game Manually

        10.2 Deep Learning Game Strategies: Generating Data

            10.2.1 Summary of the Game Strategy

            10.2.2 Simulate One Game

            10.2.3 Simulate Many Games

        10.3 Train the Deep Neural Network

            10.3.1 Preprocess the Data

            10.3.2 Train Deep Learning Game Strategies

        10.4 Play Games with the Trained Model

            10.4.1 Test One Game

            10.4.2 Test the Efficacy of the Game Strategy

        10.5 Animate the Decision-Making Process

            10.5.1 Generate Figures

            10.5.2 Create the Animation

            10.5.3 Create a Figure with Subplots

        10.6 Glossary

        10.7 Exercises

    CHAPTER 11 ◾ Deep Learning in the Cart Pole Game

        11.1 Play the Cart Pole Game in OpenAI Gym

            11.1.1 Features of the Cart Pole Game

            11.1.2 Play a Full Game

        11.2 Generate Data to Train the Model

            11.2.1 How to Define Winning and Losing?

            11.2.2 Prepare Data for the Neural Network

        11.3 Train the Deep Neural Network

            11.3.1 Preprocess the Data

            11.3.2 Train the Deep Neural Network with Data

        11.4 Play the Game with the Trained Model

            11.4.1 A best_move() Function

            11.4.2 Play One Cart Pole Game with the Trained Model

        11.5 Compare Two Games

            11.5.1 Record a Game with Random Moves

            11.5.2 Combine Frames

            11.5.3 Subplots of the Cart Pole Game Stages

        11.6 Glossary

        11.7 Exercises

    CHAPTER 12 ◾ Deep Learning in Multi-Player Games

        12.1 Create the Tic Tac Toe Game Environment

            12.1.1 Use a Python Class to Represent the Environment

            12.1.2 Create a Local Module for the Tic Tac Toe Game

            12.1.3 Verify the Custom-Made Game Environment

            12.1.4 Play a Game in the Tic Tac Toe Environment

        12.2 Train A Deep Learning Game Strategy

            12.2.1 A Blueprint of the Deep Learning Game Strategy

            12.2.2 Simulate Tic Tac Toe Games

            12.2.3 Train Your Tic Tac Toe Game Strategy

        12.3 Use the Trained Model to Play Games

            12.3.1 Best Moves Based on the Trained Model

            12.3.2 Test a Game Using the Trained Model

            12.3.3 Test the Efficacy of the Trained Model

        12.4 Animate the Deep Learning Process

            12.4.1 Probabilities of Winning for Each Hypothetical Move

            12.4.2 Animate the Whole Game

            12.4.3 Animate the Decision Making

            12.4.4 Animate Board Positions and the Decision Making

            12.4.5 Subplots of the Decision-Making Process

        12.5 Glossary

        12.6 Exercises

    CHAPTER 13 ◾ Deep Learning in Connect Four

        13.1 Create A Connect Four Game Environment

            13.1.1 A Connect Four Game Environment

            13.1.2 Verify the Connect Four Game Environment

            13.1.3 Play a Connect Four Game

        13.2 Train A Deep Neural Network

            13.2.1 The Game Plan

            13.2.2 Simulate Connect Four Games

            13.2.3 Train the Connect Four Game Strategy

        13.3 Use the Trained Model to Play Connect Four

            13.3.1 Best Moves

            13.3.2 Test Connect Four Deep Learning Game Strategies

        13.4 Animate Deep Learning in Connect Four

            13.4.1 Print Out Probabilities of Winning for Each Next Move

            13.4.2 Animate a Complete Connect Four Game

            13.4.3 Animate the Decision-Making Process

            13.4.4 Combine Board Positions and Decision Making

            13.4.5 Create Subplots of Deep Learning

        13.5 Glossary

        13.6 Exercises

SECTION V Reinforcement Learning

    CHAPTER 14 ◾ Introduction to Reinforcement Learning

        14.1 Basics of Reinforcement Learning

            14.1.1 Basic Concepts

            14.1.2 The Bellman Equation and Q-Learning

        14.2 Use Q-Values to Play the Frozen Lake Game

            14.2.1 The Logic Behind Q-Learning

            14.2.2 A Q-Table to Win the Frozen Lake Game

        14.3 Train the Q-Values

            14.3.1 What is Q-Learning?

            14.3.2 Let the Learning Begin

        14.4 Q-Learning in A Self-Made Game Environment

            14.4.1 A Self-Made Frozen Lake Game Environment

            14.4.2 Use the Q-Table in the Self-Made Game Environment

        14.5 Animate the Q-Learning Process

            14.5.1 Highlight Values and Actions in the Q-Table

            14.5.2 Animate the Use of the Q-Table

            14.5.3 Game Board Positions and Best Actions

            14.5.4 Subplots of the Q-Learning Process

        14.6 Glossary

        14.7 Exercises

    CHAPTER 15 ◾ Q-Learning with Continuous States

        15.1 The Mountain Car Game Environment

            15.1.1 The Mountain Car Game

            15.1.2 Convert a Continuous State into Discrete Values

            15.1.3 The Reward Structure of the Game

        15.2 Q-Learning in the Mountain Car Game

            15.2.1 How to Train the Q-Table

            15.2.2 Update the Q-Table

            15.2.3 Train the Q-Table via Trial and Error

        15.3 Test the Trained Q-Table

            15.3.1 Define the Test_Q() Function

            15.3.2 The Effectiveness of the Trained Q-Table

        15.4 Animate the Game Before and After Q-Learning

            15.4.1 The Mountain Car Game without Q-Learning

            15.4.2 The Mountain Car Game with Q-Learning

            15.4.3 The Mountain Car Game with and without Q-Learning

        15.5 Glossary

        15.6 Exercises

    CHAPTER 16 ◾ Solving Real-World Problems with Machine Learning

        16.1 Create A Delivery Route Game Environment

            16.1.1 Draw Delivery Routes

            16.1.2 Create a Game Environment

            16.1.3 Use the Delivery Route Game Environment

        16.2 Train A Q-Table between Any Two Positions

            16.2.1 Create and Train A Q-table

            16.2.2 Test the Trained Tabular Q-Values

        16.3 Train the Q-Table for All Possible Routes

            16.3.1 Train the Large Q-Table

            16.3.2 Test the Large Q-Table

        16.4 The Shortest Delivery Route to Eight Households

            16.4.1 Find All Possible Permutations in Python

            16.4.2 The Total Distance to Deliver to Eight Households

            16.4.3 The Shortest Route

        16.5 Animate the Delivery Route

            16.5.1 Create a Graph at Each Stop

            16.5.2 Animate the Shortest Route

            16.5.3 Subplots of the Eight Deliveries

        16.6 Glossary

        16.7 Exercises

SECTION VI Deep Reinforcement Learning

    CHAPTER 17 ◾ Deep Q-Learning

        17.1 Deep Q-Learning for the Cart Pole Game

            17.1.1 Create a Deep Q-Network

            17.1.2 Train the Deep Q-Network

        17.2 Test the Trained Deep Q-Network

            17.2.1 Test and Record One Game

            17.2.2 Test the Efficacy of the Deep Q-Network

        17.3 Animate Deep Q-Learning

            17.3.1 Draw the Current Game State and Q-Values

            17.3.2 Create A Graph for Each Time Step

        17.4 An Animation and A Picture with Subplots

        17.5 Glossary

        17.6 Exercises

    CHAPTER 18 ◾ Policy-Based Deep Reinforcement Learning

        18.1 Policy-Based Reinforcement Learning

            18.1.1 What is a Policy?

            18.1.2 What is the Policy Gradient Method?

        18.2 Get Started with Atari Games

            18.2.1 The Pong Game

            18.2.2 Preprocess the Game Pictures

            18.2.3 Use the Difference of Game Windows

        18.3 Train the Policy Gradient Agent

            18.3.1 Create a Policy Network

            18.3.2 Train the Model

        18.4 Test the Policy Gradient Agent

        18.5 Animate the Pong Games

            18.5.1 Record Games with Random Moves

            18.5.2 Combine the Animations

            18.5.3 Subplots of the Policy Gradient Agent

        18.6 Glossary

        18.7 Exercises

    CHAPTER 19 ◾ The Policy Gradient Method in Breakout

        19.1 Get Started with the Breakout Game

            19.1.1 The Breakout Game

            19.1.2 Preprocess the Game Frames

            19.1.3 Obtain the Difference of Two Game Windows

        19.2 Train the Policy Gradient Model in Breakout

            19.2.1 Changes Needed

            19.2.2 Create a Policy Network

            19.2.3 Train the Policy Gradient Agent in Breakout

        19.3 Test the Policy Gradient Agent in Breakout

            19.3.1 Test the Trained Policy Gradient Agent

            19.3.2 Search for Successful Episodes

        19.4 Zero in on Interesting Time Steps

            19.4.1 Animate Interesting Time Steps

            19.4.2 Subplots of the Interesting Time Steps

        19.5 Exercises

    CHAPTER 20 ◾ Double Deep Q-Learning

        20.1 Get Started with OpenAI Baselines

            20.1.1 The Breakout Game with OpenAI Baselines

            20.1.2 Preprocessed Frames from Baselines

            20.1.3 Subplots of Preprocessed Frames

        20.2 Train the Double Deep Q Agent

            20.2.1 Create a Double Deep Q-Network

            20.2.2 Train the Deep Q Network

        20.3 Test the Trained Breakout Agent

            20.3.1 Testing One Original Episode

            20.3.2 Play Multiple Games and Test the Average Score

        20.4 Animate Interesting Time Steps

            20.4.1 Collect a Successful Episode

            20.4.2 A Picture with Subplots

        20.5 Glossary

        20.6 Exercises

    CHAPTER 21 ◾ Space Invaders with Double Deep Q-Learning

        21.1 Getting Started with Space Invaders

            21.1.1 Space Invaders in OpenAI Gym

            21.1.2 Space Invaders with the Baselines Game Wrapper

            21.1.3 Preprocessed Space Invaders Game Windows

        21.2 Train the Double Deep Q-Network

            21.2.1 The Same Double Deep Q-Network

            21.2.2 The Same Training Process

        21.3 Test the Trained Agent in Space Invaders

            21.3.1 Testing One Full Original Episode

            21.3.2 Average Performance of the Trained Model

        21.4 Animate Space Invaders

            21.4.1 Collect Space Invaders Episodes

            21.4.2 Zero in on the Interesting Time Steps

            21.4.3 Subplots of Space Invaders

        21.5 Exercises

    CHAPTER 22 ◾ Scaling Up Double Deep Q-Learning

        22.1 Get Started with the Seaquest Game

            22.1.1 The Seaquest Game in OpenAI Gym

            22.1.2 Seaquest with the Baselines Game Wrapper

            22.1.3 Preprocessed Seaquest Game Windows

            22.1.4 Subplots of Seaquest Game Windows

        22.2 Get Started with Beam Rider

            22.2.1 Beam Rider without the Game Wrapper

            22.2.2 Beam Rider with the Baselines Game Wrapper

            22.2.3 Preprocessed Beam Rider Game Windows

            22.2.4 Subplots of Beam Rider Game Windows

        22.3 Scaling Up the Double Deep Q-Network

            22.3.1 Differences among Atari Games

            22.3.2 A Generic Double Deep Q-Network

            22.3.3 The Training Process for any Atari Game

        22.4 Try It on Seaquest

            22.4.1 Train the Model in Seaquest

            22.4.2 Test the Average Score in Seaquest

            22.4.3 Animate a Successful Episode

        22.5 Try It on Beam Rider

            22.5.1 Train the Model in Beam Rider

            22.5.2 The Average Score in Beam Rider

            22.5.3 A Successful Episode in Beam Rider

        22.6 Exercises

Bibliography

Index