Machine Learning Engineering in Action

Machine Learning Engineering in Action
brief contents
contents
preface
acknowledgments
about this book
	Who should read this book
	How this book is organized: A road map
	About the code
	liveBook discussion forum
about the author
about the cover illustration
Part 1—An introduction to machine learning engineering
	1 What is a machine learning engineer?
		1.1 Why ML engineering?
		1.2 The core tenets of ML engineering
			1.2.1 Planning
			1.2.2 Scoping and research
			1.2.3 Experimentation
			1.2.4 Development
			1.2.5 Deployment
			1.2.6 Evaluation
		1.3 The goals of ML engineering
		Summary
	2 Your data science could use some engineering
		2.1 Augmenting a complex profession with processes to increase project success
		2.2 A foundation of simplicity
		2.3 Co-opting principles of Agile software engineering
			2.3.1 Communication and cooperation
			2.3.2 Embracing and expecting change
		2.4 The foundation of ML engineering
		Summary
	3 Before you model: Planning and scoping a project
		3.1 Planning: You want me to predict what?!
			3.1.1 Basic planning for a project
			3.1.2 That first meeting
			3.1.3 Plan for demos—lots of demos
			3.1.4 Experimentation by solution building: Wasting time for pride’s sake
		3.2 Experimental scoping: Setting expectations and boundaries
			3.2.1 What is experimental scoping?
			3.2.2 Experimental scoping for the ML team: Research
			3.2.3 Experimental scoping for the ML team: Experimentation
		Summary
	4 Before you model: Communication and logistics of projects
		4.1 Communication: Defining the problem
			4.1.1 Understanding the problem
			4.1.2 Setting critical discussion boundaries
		4.2 Don’t waste our time: Meeting with cross-functional teams
			4.2.1 Experimental update meeting: Do we know what we’re doing here?
			4.2.2 SME review/prototype review: Can we solve this?
			4.2.3 Development progress review(s): Is this thing going to work?
			4.2.4 MVP review: Did you build what we asked for?
			4.2.5 Preproduction review: We really hope we didn’t screw this up
		4.3 Setting limits on your experimentation
			4.3.1 Set a time limit
			4.3.2 Can you put this into production? Would you want to maintain it?
			4.3.3 TDD vs. RDD vs. PDD vs. CDD for ML projects
		4.4 Planning for business rules chaos
			4.4.1 Embracing chaos by planning for it
			4.4.2 Human-in-the-loop design
			4.4.3 What’s your backup plan?
		4.5 Talking about results
		Summary
	5 Experimentation in action: Planning and researching an ML project
		5.1 Planning experiments
			5.1.1 Perform basic research and planning
			5.1.2 Forget the blogs—read the API docs
			5.1.3 Draw straws for an internal hackathon
			5.1.4 Level the playing field
		5.2 Performing experimental prep work
			5.2.1 Performing data analysis
			5.2.2 Moving from script to reusable code
			5.2.3 One last note on building reusable code for experimentation
		Summary
	6 Experimentation in action: Testing and evaluating a project
		6.1 Testing ideas
			6.1.1 Setting guidelines in code
			6.1.2 Running quick forecasting tests
		6.2 Whittling down the possibilities
			6.2.1 Evaluating prototypes properly
			6.2.2 Making a call on the direction to go in
			6.2.3 So . . . what’s next?
		Summary
	7 Experimentation in action: Moving from prototype to MVP
		7.1 Tuning: Automating the annoying stuff
			7.1.1 Tuning options
			7.1.2 Hyperopt primer
			7.1.3 Using Hyperopt to tune a complex forecasting problem
		7.2 Choosing the right tech for the platform and the team
			7.2.1 Why Spark?
			7.2.2 Handling tuning from the driver with SparkTrials
			7.2.3 Handling tuning from the workers with a pandas_udf
			7.2.4 Using new paradigms for teams: Platforms and technologies
		Summary
	8 Experimentation in action: Finalizing an MVP with MLflow and runtime optimization
		8.1 Logging: Code, metrics, and results
			8.1.1 MLflow tracking
			8.1.2 Please stop printing and log your information
			8.1.3 Version control, branch strategies, and working with others
		8.2 Scalability and concurrency
			8.2.1 What is concurrency?
			8.2.2 What you can (and can’t) run asynchronously
		Summary
Part 2—Preparing for production: Creating maintainable ML
	9 Modularity for ML: Writing testable and legible code
		9.1 Understanding monolithic scripts and why they are bad
			9.1.1 How monoliths come into being
			9.1.2 Walls of text
			9.1.3 Considerations for monolithic scripts
		9.2 Debugging walls of text
		9.3 Designing modular ML code
		9.4 Using test-driven development for ML
		Summary
	10 Standards of coding and creating maintainable ML code
		10.1 ML code smells
		10.2 Naming, structure, and code architecture
			10.2.1 Naming conventions and structure
			10.2.2 Trying to be too clever
			10.2.3 Code architecture
		10.3 Tuple unpacking and maintainable alternatives
			10.3.1 Tuple unpacking example
			10.3.2 A solid alternative to tuple unpacking
		10.4 Blind to issues: Eating exceptions and other bad practices
			10.4.1 Try/catch with the precision of a shotgun
			10.4.2 Exception handling with laser precision
			10.4.3 Handling errors the right way
		10.5 Use of global mutable objects
			10.5.1 How mutability can burn you
			10.5.2 Encapsulation to prevent mutable side effects
		10.6 Excessively nested logic
		Summary
	11 Model measurement and why it’s so important
		11.1 Measuring model attribution
			11.1.1 Measuring prediction performance
			11.1.2 Clarifying correlation vs. causation
		11.2 Leveraging A/B testing for attribution calculations
			11.2.1 A/B testing 101
			11.2.2 Evaluating continuous metrics
			11.2.3 Using alternative displays and tests
			11.2.4 Evaluating categorical metrics
		Summary
	12 Holding on to your gains by watching for drift
		12.1 Detecting drift
			12.1.1 What influences drift?
		12.2 Responding to drift
			12.2.1 What can we do about it?
			12.2.2 Responding to drift
		Summary
	13 ML development hubris
		13.1 Elegant complexity vs. overengineering
			13.1.1 Lightweight scripted style (imperative)
			13.1.2 An overengineered mess
		13.2 Unintentional obfuscation: Could you read this if you didn’t write it?
			13.2.1 The flavors of obfuscation
			13.2.2 Troublesome coding habits recap
		13.3 Premature generalization, premature optimization, and other bad ways to show how smart you are
			13.3.1 Generalization and frameworks: Avoid them until you can’t
			13.3.2 Optimizing too early
		13.4 Do you really want to be the canary? Alpha testing and the dangers of the open source coal mine
		13.5 Technology-driven development vs. solution-driven development
		Summary
Part 3—Developing production machine learning code
	14 Writing production code
		14.1 Have you met your data?
			14.1.1 Make sure you have the data
			14.1.2 Check your data provenance
			14.1.3 Find a source of truth and align on it
			14.1.4 Don’t embed data cleansing into your production code
		14.2 Monitoring your features
		14.3 Monitoring everything else in the model life cycle
		14.4 Keeping things as simple as possible
			14.4.1 Simplicity in problem definitions
			14.4.2 Simplicity in implementation
		14.5 Wireframing ML projects
		14.6 Avoiding cargo cult ML behavior
		Summary
	15 Quality and acceptance testing
		15.1 Data consistency
			15.1.1 Training and inference skew
			15.1.2 A brief intro to feature stores
			15.1.3 Process over technology
			15.1.4 The dangers of a data silo
		15.2 Fallbacks and cold starts
			15.2.1 Leaning heavily on prior art
			15.2.2 Cold-start woes
		15.3 End user vs. internal use testing
			15.3.1 Biased testing
			15.3.2 Dogfooding
			15.3.3 SME evaluation
		15.4 Model interpretability
			15.4.1 Shapley additive explanations
			15.4.2 Using shap
		Summary
	16 Production infrastructure
		16.1 Artifact management
			16.1.1 MLflow’s model registry
			16.1.2 Interfacing with the model registry
		16.2 Feature stores
			16.2.1 What a feature store is used for
			16.2.2 Using a feature store
			16.2.3 Evaluating a feature store
		16.3 Prediction serving architecture
			16.3.1 Determining serving needs
			16.3.2 Bulk external delivery
			16.3.3 Microbatch streaming
			16.3.4 Real-time server-side
			16.3.5 Integrated models (edge deployment)
		Summary
Appendix A—Big O(no) and how to think about runtime performance
	A.1 What is Big O, anyway?
		A.1.1 A gentle introduction to complexity
	A.2 Complexity by example
		A.2.1 O(1): The “It doesn’t matter how big the data is” algorithm
		A.2.2 O(n): The linear relationship algorithm
		A.2.3 O(n2): A polynomial relationship to the size of the collection
	A.3 Analyzing decision-tree complexity
	A.4 General algorithmic complexity for ML
Appendix B—Setting up a development environment
	B.1 The case for a clean experimentation environment
	B.2 Containers to deal with dependency hell
	B.3 Creating a container-based pristine environment for experimentation
index
	A
	B
	C
	D
	E
	F
	G
	H
	I
	J
	K
	L
	M
	N
	O
	P
	Q
	R
	S
	T
	U
	V
	W
	X