Hyperparameter Tuning for Machine and Deep Learning with R: A Practical Guide

Foreword
Contents
Contributors
Abbreviations
1 Introduction
Part I Theory
2 Tuning: Methodology
	2.1 Introduction to Hyperparameter Tuning
	2.2 Performance Measures for Hyperparameter Tuning
		2.2.1 Metrics
		2.2.2 Performance Measures
		2.2.3 Measures for Classification
		2.2.4 Measures for Regression
	2.3 Hyperparameter Tuning
	2.4 Model Selection and Assessment
	2.5 Tunability and Complexity
	2.6 The Basic HPT Process
	2.7 Practical Considerations
		2.7.1 Some Thoughts on Cross Validation
		2.7.2 Replicability and Stochasticity
		2.7.3 Implementation in R
3 Models
	3.1 Methods and Hyperparameters
	3.2 k-Nearest Neighbor
		3.2.1 Description
		3.2.2 Hyperparameters of k-Nearest Neighbor
	3.3 Regularized Regression (Elastic Net)
		3.3.1 Description
		3.3.2 Hyperparameters of Elastic Net
	3.4 Decision Trees
		3.4.1 Description
		3.4.2 Hyperparameters of Decision Trees
	3.5 Random Forest
		3.5.1 Description
		3.5.2 Hyperparameters of Random Forests
	3.6 Gradient Boosting (xgboost)
		3.6.1 Description
		3.6.2 Hyperparameters of Gradient Boosting
	3.7 Support Vector Machines
		3.7.1 Description
		3.7.2 Hyperparameters of the SVM
	3.8 Deep Neural Networks
		3.8.1 Description
		3.8.2 Hyperparameters of Deep Neural Networks
	3.9 Summary and Discussion
4 Hyperparameter Tuning Approaches
	4.1 Hyperparameter Tuning: Approaches and Goals
	4.2 Special Case: Monotonous Hyperparameters
	4.3 Model-Free Search
		4.3.1 Manual Search
		4.3.2 Undirected Search
		4.3.3 Directed Search
	4.4 Model-Based Search
	4.5 Sequential Parameter Optimization Toolbox
		4.5.1 spot as an Optimizer
		4.5.2 spot's Initial Phase
		4.5.3 The Function spotLoop
		4.5.4 Entering the Main Loop
		4.5.5 Final Steps
	4.6 Kriging
		4.6.1 The Kriging Model
		4.6.2 Kriging Prediction
		4.6.3 Expected Improvement
		4.6.4 Infill Criteria with Noisy Data
		4.6.5 spot's Workhorse: Kriging
		4.6.6 krigingLikelihood
		4.6.7 Predictions
	4.7 Program Code
5 Ranking and Result Aggregation
	5.1 Comparing Algorithms
	5.2 Ranking
	5.3 Rank Aggregation
	5.4 Result Analysis
	5.5 Statistical Inference
	5.6 Definitions
		5.6.1 Hypothesis Testing
		5.6.2 Power
		5.6.3 p-Value
		5.6.4 Effect Size
		5.6.5 Sample Size Determination and Power Calculations
		5.6.6 Issues
	5.7 Severity
		5.7.1 Motivation
		5.7.2 Severity: Definition
		5.7.3 Two Examples
		5.7.4 Discussion of the 80% Threshold
		5.7.5 A Comment on the Normality Assumption
	5.8 Severity: Application in Benchmarking
		5.8.1 Experiment I: Rosenbrock
		5.8.2 Experiment II: Freudenstein-Roth
		5.8.3 Experiment III: Powell's Badly Scaled Test Function
	5.9 Summary and Discussion
Part II Applications
6 Hyperparameter Tuning and Optimization Applications
	6.1 Surrogate Optimization
	6.2 Hyperparameter Tuning in Machine and Deep Learning
	6.3 HPT Software Tools
	6.4 Summary and Discussion
7 Hyperparameter Tuning in German Official Statistics
	7.1 Official Statistics
	7.2 Machine Learning in Official Statistics
	7.3 Challenges in Tuning for the Federal Statistical Office
	7.4 Dealing with the Challenges
8 Case Study I: Tuning Random Forest (Ranger)
	8.1 Introduction
	8.2 Data Description
		8.2.1 The Census Data Set
		8.2.2 getDataCensus: Getting the Data from OML
	8.3 Experimental Setup and Configuration of the Random Forest Model
		8.3.1 getMlConfig: Configuration of the ML Models
		8.3.2 Implementation Details: getMlConfig
	8.4 Objective Function (Model Performance)
		8.4.1 Performance Measures
		8.4.2 Handling Errors
		8.4.3 Imputation of Missing Data
		8.4.4 getObjf: The Objective Function
	8.5 spot: Experimental Setup for the Hyperparameter Tuner
	8.6 Tunability
		8.6.1 Progress
		8.6.2 evalParamCensus: Comparing Default and Tuned Parameters on Test Data
	8.7 Analyzing the Random Forest Tuning Process
	8.8 Severity: Validating the Results
	8.9 Summary and Discussion
	8.10 Program Code
9 Case Study II: Tuning of Gradient Boosting (xgboost)
	9.1 Introduction
	9.2 Data Description
	9.3 getMlConfig: Experimental Setup and Configuration of the Gradient Boosting Model
		9.3.1 getMlrTask: Problem Design and Definition of the Machine Learning Task
		9.3.2 getModelConf Algorithm Design—Hyperparameters of the Models
		9.3.3 getMlrResample: Training and Test Data
	9.4 Objective Function (Model Performance)
	9.5 spot: Experimental Setup for the Hyperparameter Tuner
	9.6 Tunability
		9.6.1 Progress
		9.6.2 evalParamCensus: Comparing Default and Tuned Parameters on Test Data
	9.7 Analyzing the Gradient Boosting Tuning Process
	9.8 Severity: Validating the Results
	9.9 Summary and Discussion
	9.10 Program Code
10 Case Study III: Tuning of Deep Neural Networks
	10.1 Introduction
	10.2 Data Description
		10.2.1 getDataCensus: Getting the Data from OpenML
		10.2.2 getGenericTrainValTestData: Split Data in Train, Validation, and Test Data
		10.2.3 genericDataPrep: Spec
	10.3 Experimental Setup and Configuration of the Deep Learning Models
		10.3.1 getKerasConf: keras and Tensorflow Configuration
		10.3.2 getModelConf: DL Hyperparameters
		10.3.3 The Neural Network
	10.4 funKerasGeneric: The Objective Function
	10.5 spot: Experimental Setup for the Hyperparameter Tuner
	10.6 Tunability
		10.6.1 Progress
		10.6.2 evalParamCensus: Comparing Default and Tuned Parameters on Test Data
	10.7 Analysing the Deep Learning Tuning Process
	10.8 Severity: Validating the Results
	10.9 Summary and Discussion
	10.10 Program Code
11 Case Study IV: Tuned Reinforcement Learning (in Python)
	11.1 Introduction
	11.2 Materials and Methods
		11.2.1 Software
		11.2.2 Task Environment: Inverted Pendulum
		11.2.3 Learning Algorithm
	11.3 Setting up the Tuning Experiment
		11.3.1 File: run.py
		11.3.2 Tuned Parameters
		11.3.3 Further Configuration of SPOT
		11.3.4 Post-processing and Validating the Results
	11.4 Results
	11.5 Severity: Validating the Results
	11.6 Summary and Discussion
12 Global Study: Influence of Tuning
	12.1 Introduction
	12.2 Research Questions
	12.3 Setup
		12.3.1 Model Configuration
		12.3.2 Runtime For the Global Study
		12.3.3 Benchmark of Runtime
		12.3.4 Defaults and Replications
	12.4 Results
	12.5 Discussion
		12.5.1 Rank-Analysis
		12.5.2 Rank-Analysis: Classification
		12.5.3 Rank-Analysis: Regression
		12.5.4 Problem Analysis: Difficulty
	12.6 Summary and Discussion
Appendix Software Installations
A.1 Installing SPOT and SPOTMisc
A.2 Installing Python
A.2.1 Create and Activate a Python Environment in the Project
A.2.2 Install Python Packages in the Environment
A.2.3 Install and Configure Reticulate to Use the Correct Python Version
A.3 Installing Keras
A.4 Installing SPOT
A.5 Installation on ARM Macs
Appendix  References
Index