Machine Learning, Animated

Cover
Half Title
Series Page
Title Page
Copyright Page
Dedication
Contents
Preface
Acknowledgments
SECTION I: Installing Python and Learning Animations
	CHAPTER 1: Installing Anaconda and Jupyter Notebook
		1.1. WHY PYTHON FOR MACHINE LEARNING?
			1.1.1. The Rise of Python
			1.1.2. Python for Machine Learning
		1.2. INSTALLING ANACONDA
			1.2.1. Installing Anaconda in Windows
			1.2.2. Installing Anaconda in macOS
			1.2.3. Installing Anaconda in Linux
			1.2.4. Difference between Conda-install and Pip-install
		1.3. VIRTUAL ENVIRONMENT FOR THIS BOOK
			1.3.1. Create the Virtual Environment MLA
			1.3.2. Activate the Virtual Environment
			1.3.3. De-activate the Virtual Environment
		1.4. SET UP JUPYTER NOTEBOOK IN THE VIRTUAL ENVIRONMENT
			1.4.1. Write Python in Jupyter Notebook
			1.4.2. Issue Commands in Jupyter Notebook
		1.5. FILE SYSTEM FOR THE BOOK
		1.6. GLOSSARY
		1.7. EXERCISES
	CHAPTER 2: Creating Animations
		2.1. CREATE PLOTS WITH MATPLOTLIB
			2.1.1. A Single Line Plot
			2.1.2. Multiple Lines in the Same Plot
		2.2. CREATE SUBPLOTS
			2.2.1. Create Individual Plots
			2.2.2. Create Subplots
		2.3. CREATE ANIMATED PLOTS
			2.3.1. Generate Annual Line Plots
			2.3.2. Animate the Plots
		2.4. CREATE ANIMATED BAR CHARTS
			2.4.1. Create a Horizontal Bar Chart
			2.4.2. Generate Annual Bar Charts
			2.4.3. Animate the Bar Charts
		2.5. PUT BAR CHARTS AND PLOTS SIDE BY SIDE
			2.5.1. Combine a Bar Chart and a Plot
			2.5.2. Create an Animation of the Combined Pictures
		2.6. ANIMATED PIE CHARTS
			2.6.1. Create a Pie Chart
			2.6.2. Generate Annual Pie Charts
			2.6.3. Animate the Combined Pie Charts and Plots
		2.7. APPENDIX: DOWNLOAD AND CLEAN UP THE GDP DATA
		2.8. GLOSSARY
		2.9. EXERCISES
SECTION II: Machine Learning Basics
	CHAPTER 3: Machine Learning: An Overview
		3.1. ML: A NEW PARADIGM FOR AI
			3.1.1. What is AI?
			3.1.2. Rule-Based AI
			3.1.3. What is ML? Why ML Matters?
			3.1.4. Why is ML so Popular?
		3.2. DIFFERENT TYPES OF ML
			3.2.1. Supervised Learning
			3.2.2. Unsupervised Learning
			3.2.3. Reinforcement Learning
		3.3. DEEP REINFORCEMENT LEARNING
			3.3.1. Deep Learning
			3.3.2. Combine Deep Learning and Reinforcement Learning
		3.4. APPLY ML IN THE REAL WORLD
			3.4.1. The Delivery Route Problem
			3.4.2. Try the Problem before You Know the Answer
		3.5. GLOSSARY
		3.6. EXERCISES
	CHAPTER 4: Gradient Descent – Where Magic Happens
		4.1. OPTIMIZATION THROUGH GRID SEARCH
			4.1.1. How Grid Search Achieves Optimization
			4.1.2. Curse of Dimensionality and Directional Grid Search
		4.2. GRADIENT DESCENT
		4.3. USE TENSORFLOW TO CALCULATE GRADIENTS
			4.3.1. Install TensorFlow
			4.3.2. Calculate Gradients Using TensorFlow
			4.3.3. Gradient Descent Optimization with TensorFlow
			4.3.4. Animate the Optimization Process
		4.4. CHOOSE THE RIGHT LEARNING RATE
			4.4.1. When the Learning Rate is Too Large
			4.4.2. When the Learning Rate is Too Small
		4.5. COMPARE LEARNING RATES
			4.5.1. Combine Animations
			4.5.2. Subplots of Different Stages
		4.6. GLOSSARY
		4.7. EXERCISES
	CHAPTER 5: Introduction to Neural Networks
		5.1. ANATOMY OF A NEURAL NETWORK
			5.1.1. Elements of a Neural Network
			5.1.2. How Does a Neural Network Learn?
			5.1.3. Make Predictions
		5.2. ANIMATE THE LEARNING PROCESS
			5.2.1. Generate Graphs
			5.2.2. Create Animation Based on Graphs
			5.2.3. Subplots of Different Stages
		5.3. CREATE A NEURAL NETWORK WITH KERAS
			5.3.1. Construct the Model
			5.3.2. Compile and Train the Model
			5.3.3. Make Predictions
		5.4. CUSTOMIZE TRAINING WITH GRADIENTTAPE
			5.4.1. Construct the Model
			5.4.2. Train the Model
			5.4.3. Make Predictions
		5.5. GLOSSARY
		5.6. EXERCISES
	CHAPTER 6: Activation Functions
		6.1. WHY DO WE NEED ACTIVATION FUNCTIONS?
			6.1.1. Construct a Neural Network
			6.1.2. Learn a Nonlinear Relation without Activation
		6.2. THE RELU ACTIVATION FUNCTION
			6.2.1. What is ReLU?
			6.2.2. Animate the ReLU Function
			6.2.3. Use ReLU to Model Nonlinearity
		6.3. THE SIGMOID ACTIVATION FUNCTION
			6.3.1. Plot the Sigmoid Function
			6.3.2. Animate the Sigmoid Function
			6.3.3. Combine Animations
			6.3.4. A Picture with Subplots of Different Stages
		6.4. THE SOFTMAX ACTIVATION FUNCTION
			6.4.1. What is the Softmax Function?
			6.4.2. A Diagram of the Softmax Function
		6.5. GLOSSARY
		6.6. EXERCISES
SECTION III: Binary and Multi-Category Classifications
	CHAPTER 7: Binary Classifications
		7.1. WHAT IS A BINARY CLASSIFICATION PROBLEM
			7.1.1. Sigmoid Activation in Binary Classifications
			7.1.2. The Binary Cross-Entropy Loss Function
		7.2. PROCESS IMAGE DATA
			7.2.1. Download Data
			7.2.2. Convert NumPy Arrays to Pictures and Back
			7.2.3. Match Pictures with Labels
		7.3. BINARY CLASSIFICATION WITH A LOGIT REGRESSION
			7.3.1. Prepare the Data
			7.3.2. Train the Logit Model
			7.3.3. Predict Using the Logit Model
		7.4. BINARY CLASSIFICATION WITH A SIMPLE NEURAL NETWORK
			7.4.1. Train and Test Using a Neural Network
			7.4.2. Focus on Two Examples
			7.4.3. Diagrams of the Network and Predictions
			7.4.4. Animate the Training Process
			7.4.5. Animate the Predictions for the Deer
		7.5. COMBINE THE ANIMATIONS
			7.5.1. Animate the Two Predictions
			7.5.2. Subplots
		7.6. BINARY CLASSIFICATION WITH A DEEP NEURAL NETWORK
		7.7. APPENDIX: LOAD CIFAR10 FROM TENSORFLOW DIRECTLY
		7.8. GLOSSARY
		7.9. EXERCISES
	CHAPTER 8: Convolutional Neural Networks
		8.1. WHAT ARE CONVOLUTIONAL NEURAL NETWORKS (CNNS)?
			8.1.1. Our Running Example
			8.1.2. A Horizontal Filter
		8.2. CONVOLUTION OPERATIONS
			8.2.1. Calculations in a Convolution Operation
			8.2.2. Animate the Convolution Operations
			8.2.3. Subplots
		8.3. STRIDE AND PADDING
			8.3.1. A Filter without Padding and a Stride of 2
			8.3.2. Animate the Diagonal Filter
			8.3.3. Animate the Diagonal Filter Convolution Operation
			8.3.4. Subplots for Strides
		8.4. COMBINE THE TWO ANIMATIONS
			8.4.1. Combine the Animations
		8.5. MAX POOLING
		8.6. BINARY CLASSIFICATIONS WITH CONVOLUTIONAL LAYERS
		8.7. GLOSSARY
		8.8. EXERCISES
	CHAPTER 9: Multi-Category Image Classifications
		9.1. IMAGE AUGMENTATIONS
			9.1.1. The Keras Image Generator
			9.1.2. Visualize Image Augmentations
		9.2. WHAT IS MULTI-CATEGORY CLASSIFICATION?
			9.2.1. One-Hot Encoder for Labels
			9.2.2. The Activation and Loss Functions
		9.3. TRAIN THE MULTI-CATEGORY CLASSIFICATION MODEL
			9.3.1. Load the Full Data Set
			9.3.2. Convert Labels to One-Hot Variables
			9.3.3. Train the Model
			9.3.4. Evaluate the Model
		9.4. ANIMATE THE LEARNING PROCESS
			9.4.1. Select Example Pictures
			9.4.2. Animate Prediction Changes
			9.4.3. Subplots of the Predictions on the Truck Image
			9.4.4. Animate Predictions on the Frog Image
			9.4.5. Subplots of the Predictions on the Frog Image
			9.4.6. Combine the Animations
		9.5. GLOSSARY
		9.6. EXERCISES
SECTION IV: Developing Deep Learning Game Strategies
	CHAPTER 10: Deep Learning Game Strategies
		10.1. GET STARTED WITH THE OPENAI GYM ENVIRONMENT
			10.1.1. Basic Elements of a Game Environment
			10.1.2. The Frozen Lake Game
			10.1.3. Play the Frozen Lake Game Manually
		10.2. DEEP LEARNING GAME STRATEGIES: GENERATING DATA
			10.2.1. Summary of the Game Strategy
			10.2.2. Simulate One Game
			10.2.3. Simulate Many Games
		10.3. TRAIN THE DEEP NEURAL NETWORK
			10.3.1. Preprocess the Data
			10.3.2. Train Deep Learning Game Strategies
		10.4. PLAY GAMES WITH THE TRAINED MODEL
			10.4.1. Test One Game
			10.4.2. Test the Efficacy of the Game Strategy
		10.5. ANIMATE THE DECISION-MAKING PROCESS
			10.5.1. Generate Figures
			10.5.2. Create the Animation
			10.5.3. Create a Figure with Subplots
		10.6. GLOSSARY
		10.7. EXERCISES
	CHAPTER 11: Deep Learning in the Cart Pole Game
		11.1. PLAY THE CART POLE GAME IN OPENAI GYM
			11.1.1. Features of the Cart Pole Game
			11.1.2. Play a Full Game
		11.2. GENERATE DATA TO TRAIN THE MODEL
			11.2.1. How to Define Winning and Losing?
			11.2.2. Prepare Data for the Neural Network
		11.3. TRAIN THE DEEP NEURAL NETWORK
			11.3.1. Preprocess the Data
			11.3.2. Train the Deep Neural Network with Data
		11.4. PLAY THE GAME WITH THE TRAINED MODEL
			11.4.1. A best_move() Function
			11.4.2. Play One Cart Pole Game with the Trained Model
		11.5. COMPARE TWO GAMES
			11.5.1. Record a Game with Random Moves
			11.5.2. Combine Frames
			11.5.3. Subplots of the Cart Pole Game Stages
		11.6. GLOSSARY
		11.7. EXERCISES
	CHAPTER 12: Deep Learning in Multi-Player Games
		12.1. CREATE THE TIC TAC TOE GAME ENVIRONMENT
			12.1.1. Use a Python Class to Represent the Environment
			12.1.2. Create a Local Module for the Tic Tac Toe Game
			12.1.3. Verify the Custom-Made Game Environment
			12.1.4. Play a Game in the Tic Tac Toe Environment
		12.2. TRAIN A DEEP LEARNING GAME STRATEGY
			12.2.1. A Blueprint of the Deep Learning Game Strategy
			12.2.2. Simulate Tic Tac Toe Games
			12.2.3. Train Your Tic Tac Toe Game Strategy
		12.3. USE THE TRAINED MODEL TO PLAY GAMES
			12.3.1. Best Moves Based on the Trained Model
			12.3.2. Test a Game Using the Trained Model
			12.3.3. Test the Efficacy of the Trained Model
		12.4. ANIMATE THE DEEP LEARNING PROCESS
			12.4.1. Probabilities of Winning for Each Hypothetical Move
			12.4.2. Animate the Whole Game
			12.4.3. Animate the Decision Making
			12.4.4. Animate Board Positions and the Decision Making
			12.4.5. Subplots of the Decision-Making Process
		12.5. GLOSSARY
		12.6. EXERCISES
	CHAPTER 13: Deep Learning in Connect Four
		13.1. CREATE A CONNECT FOUR GAME ENVIRONMENT
			13.1.1. A Connect Four Game Environment
			13.1.2. Verify the Connect Four Game Environment
			13.1.3. Play a Connect Four Game
		13.2. TRAIN A DEEP NEURAL NETWORK
			13.2.1. The Game Plan
			13.2.2. Simulate Connect Four Games
			13.2.3. Train the Connect Four Game Strategy
		13.3. USE THE TRAINED MODEL TO PLAY CONNECT FOUR
			13.3.1. Best Moves
			13.3.2. Test Connect Four Deep Learning Game Strategies
		13.4. ANIMATE DEEP LEARNING IN CONNECT FOUR
			13.4.1. Print Out Probabilities of Winning for Each Next Move
			13.4.2. Animate a Complete Connect Four Game
			13.4.3. Animate the Decision-Making Process
			13.4.4. Combine Board Positions and Decision Making
			13.4.5. Create Subplots of Deep Learning
		13.5. GLOSSARY
		13.6. EXERCISES
SECTION V: Reinforcement Learning
	CHAPTER 14: Introduction to Reinforcement Learning
		14.1. BASICS OF REINFORCEMENT LEARNING
			14.1.1. Basic Concepts
			14.1.2. The Bellman Equation and Q-Learning
		14.2. USE Q-VALUES TO PLAY THE FROZEN LAKE GAME
			14.2.1. The Logic Behind Q-Learning
			14.2.2. A Q-Table to Win the Frozen Lake Game
		14.3. TRAIN THE Q-VALUES
			14.3.1. What is Q-Learning?
			14.3.2. Let the Learning Begin
		14.4. Q-LEARNING IN A SELF-MADE GAME ENVIRONMENT
			14.4.1. A Self-Made Frozen Lake Game Environment
			14.4.2. Use the Q-Table in the Self-Made Game Environment
		14.5. ANIMATE THE Q-LEARNING PROCESS
			14.5.1. Highlight Values and Actions in the Q-Table
			14.5.2. Animate the Use of the Q-Table
			14.5.3. Game Board Positions and Best Actions
			14.5.4. Subplots of the Q-Learning Process
		14.6. GLOSSARY
		14.7. EXERCISES
	CHAPTER 15: Q-Learning with Continuous States
		15.1. THE MOUNTAIN CAR GAME ENVIRONMENT
			15.1.1. The Mountain Car Game
			15.1.2. Convert a Continuous State into Discrete Values
			15.1.3. The Reward Structure of the Game
		15.2. Q-LEARNING IN THE MOUNTAIN CAR GAME
			15.2.1. How to Train the Q-Table
			15.2.2. Update the Q-Table
			15.2.3. Train the Q-Table via Trial and Error
		15.3. TEST THE TRAINED Q-TABLE
			15.3.1. Define the Test_Q() Function
			15.3.2. The Effectiveness of the Trained Q-Table
		15.4. ANIMATE THE GAME BEFORE AND AFTER Q-LEARNING
			15.4.1. The Mountain Car Game without Q-Learning
			15.4.2. The Mountain Car Game with Q-Learning
			15.4.3. The Mountain Car Game with and without Q-Learning
		15.5. GLOSSARY
		15.6. EXERCISES
	CHAPTER 16: Solving Real-World Problems with Machine Learning
		16.1. CREATE A DELIVERY ROUTE GAME ENVIRONMENT
			16.1.1. Draw Delivery Routes
			16.1.2. Create a Game Environment
			16.1.3. Use the Delivery Route Game Environment
		16.2. TRAIN A Q-TABLE BETWEEN ANY TWO POSITIONS
			16.2.1. Create and Train A Q-table
			16.2.2. Test the Trained Tabular Q-Values
		16.3. TRAIN THE Q-TABLE FOR ALL POSSIBLE ROUTES
			16.3.1. Train the Large Q-Table
			16.3.2. Test the Large Q-Table
		16.4. THE SHORTEST DELIVERY ROUTE TO EIGHT HOUSEHOLDS
			16.4.1. Find All Possible Permutations in Python
			16.4.2. The Total Distance to Deliver to Eight Households
			16.4.3. The Shortest Route
		16.5. ANIMATE THE DELIVERY ROUTE
			16.5.1. Create a Graph at Each Stop
			16.5.2. Animate the Shortest Route
			16.5.3. Subplots of the Eight Deliveries
		16.6. GLOSSARY
		16.7. EXERCISES
SECTION VI: Deep Reinforcement Learning
	CHAPTER 17: Deep Q-Learning
		17.1. DEEP Q-LEARNING FOR THE CART POLE GAME
			17.1.1. Create a Deep Q-Network
			17.1.2. Train the Deep Q-Network
		17.2. TEST THE TRAINED DEEP Q-NETWORK
			17.2.1. Test and Record One Game
			17.2.2. Test the Efficacy of the Deep Q-Network
		17.3. ANIMATE DEEP Q-LEARNING
			17.3.1. Draw the Current Game State and Q-Values
			17.3.2. Create A Graph for Each Time Step
		17.4. AN ANIMATION AND A PICTURE WITH SUBPLOTS
		17.5. GLOSSARY
		17.6. EXERCISES
	CHAPTER 18: Policy-Based Deep Reinforcement Learning
		18.1. POLICY-BASED REINFORCEMENT LEARNING
			18.1.1. What is a Policy?
			18.1.2. What is the Policy Gradient Method?
		18.2. GET STARTED WITH ATARI GAMES
			18.2.1. The Pong Game
			18.2.2. Preprocess the Game Pictures
			18.2.3. Use the Difference of Game Windows
		18.3. TRAIN THE POLICY GRADIENT AGENT
			18.3.1. Create a Policy Network
			18.3.2. Train the Model
		18.4. TEST THE POLICY GRADIENT AGENT
		18.5. ANIMATE THE PONG GAMES
			18.5.1. Record Games with Random Moves
			18.5.2. Combine the Animations
			18.5.3. Subplots of the Policy Gradient Agent
		18.6. GLOSSARY
		18.7. EXERCISES
	CHAPTER 19: The Policy Gradient Method in Breakout
		19.1. GET STARTED WITH THE BREAKOUT GAME
			19.1.1. The Breakout Game
			19.1.2. Preprocess the Game Frames
			19.1.3. Obtain the Difference of Two Game Windows
		19.2. TRAIN THE POLICY GRADIENT MODEL IN BREAKOUT
			19.2.1. Changes Needed
			19.2.2. Create a Policy Network
			19.2.3. Train the Policy Gradient Agent in Breakout
		19.3. TEST THE POLICY GRADIENT AGENT IN BREAKOUT
			19.3.1. Test the Trained Policy Gradient Agent
			19.3.2. Search for Successful Episodes
		19.4. ZERO IN ON INTERESTING TIME STEPS
			19.4.1. Animate Interesting Time Steps
			19.4.2. Subplots of the Interesting Time Steps
		19.5. EXERCISES
	CHAPTER 20: Double Deep Q-Learning
		20.1. GET STARTED WITH OPENAI BASELINES
			20.1.1. The Breakout Game with OpenAI Baselines
			20.1.2. Preprocessed Frames from Baselines
			20.1.3. Subplots of Preprocessed Frames
		20.2. TRAIN THE DOUBLE DEEP Q AGENT
			20.2.1. Create a Double Deep Q-Network
			20.2.2. Train the Deep Q Network
		20.3. TEST THE TRAINED BREAKOUT AGENT
			20.3.1. Testing One Original Episode
			20.3.2. Play Multiple Games and Test the Average Score
		20.4. ANIMATE INTERESTING TIME STEPS
			20.4.1. Collect a Successful Episode
			20.4.2. A Picture with Subplots
		20.5. GLOSSARY
		20.6. EXERCISES
	CHAPTER 21: Space Invaders with Double Deep Q-Learning
		21.1. GETTING STARTED WITH SPACE INVADERS
			21.1.1. Space Invaders in OpenAI Gym
			21.1.2. Space Invaders with the Baselines Game Wrapper
			21.1.3. Preprocessed Space Invaders Game Windows
		21.2. TRAIN THE DOUBLE DEEP Q-NETWORK
			21.2.1. The Same Double Deep Q-Network
			21.2.2. The Same Training Process
		21.3. TEST THE TRAINED AGENT IN SPACE INVADERS
			21.3.1. Testing One Full Original Episode
			21.3.2. Average Performance of the Trained Model
		21.4. ANIMATE SPACE INVADERS
			21.4.1. Collect Space Invaders Episodes
			21.4.2. Zero in on the Interesting Time Steps
			21.4.3. Subplots of Space Invaders
		21.5. EXERCISES
	CHAPTER 22: Scaling Up Double Deep Q-Learning
		22.1. GET STARTED WITH THE SEAQUEST GAME
			22.1.1. The Seaquest Game in OpenAI Gym
			22.1.2. Seaquest with the Baselines Game Wrapper
			22.1.3. Preprocessed Seaquest Game Windows
			22.1.4. Subplots of Seaquest Game Windows
		22.2. GET STARTED WITH BEAM RIDER
			22.2.1. Beam Rider without the Game Wrapper
			22.2.2. Beam Rider with the Baselines Game Wrapper
			22.2.3. Preprocessed Beam Rider Game Windows
			22.2.4. Subplots of Beam Rider Game Windows
		22.3. SCALING UP THE DOUBLE DEEP Q-NETWORK
			22.3.1. Differences among Atari Games
			22.3.2. A Generic Double Deep Q-Network
			22.3.3. The Training Process for any Atari Game
		22.4. TRY IT ON SEAQUEST
			22.4.1. Train the Model in Seaquest
			22.4.2. Test the Average Score in Seaquest
			22.4.3. Animate a Successful Episode
		22.5. TRY IT ON BEAM RIDER
			22.5.1. Train the Model in Beam Rider
			22.5.2. The Average Score in Beam Rider
			22.5.3. A Successful Episode in Beam Rider
		22.6. EXERCISES
Bibliography
Index