Bash for Data Scientists

Bash for Data Scientists
	CONTENTS
	PREFACE
		WHAT IS THE GOAL?
		IS THIS BOOK IS FOR ME AND WHAT WILL I LEARN?
		HOW WERE THE CODE SAMPLES CREATED?
		WHAT YOU NEED TO KNOW FOR THIS BOOK
		WHICH BASH COMMANDS ARE EXCLUDED?
		HOW DO I SET UP A COMMAND SHELL?
		WHAT ARE THE “NEXT STEPS” AFTER FINISHING THIS BOOK?
CHAPTER 1 INTRODUCTION
	WHAT IS UNIX?
		Available Shell Types
	WHAT IS BASH?
		Getting Help for Bash Commands
		Navigating Around Directories
		The history Command
	LISTING FILENAMES WITH THE LS COMMAND
	DISPLAYING CONTENTS OF FILES
		The cat Command
		The head and tail Commands
		The Pipe Symbol
		The fold Command
	FILE OWNERSHIP: OWNER, GROUP, AND WORLD
	HIDDEN FILES
	HANDLING PROBLEMATIC FILENAMES
	WORKING WITH ENVIRONMENT VARIABLES
		The env Command
		Useful Environment Variables
		Setting the PATH Environment Variable
		Specifying Aliases and Environment Variables
	FINDING EXECUTABLE FILES
	THE printf COMMAND AND THE echo COMMAND
	THE cut COMMAND
	THE echo COMMAND AND WHITESPACES
	COMMAND SUBSTITUTION (“BACK TICK”)
	THE PIPE SYMBOL AND MULTIPLE COMMA
	USING A SEMICOLON TO SEPARATE COMMANDS
	THE paste COMMAND
		Inserting Blank Lines with the paste Command
	A SIMPLE USE CASE WITH THE paste COMMAND
	A SIMPLE USE CASE WITH cut AND paste COMMANDS
	WORKING WITH META CHARACTERS
	WORKING WITH CHARACTER CLASSES
	WHAT ABOUT ZSH?
		Switching between bash and zsh
		Configuring zsh
	SUMMARY
CHAPTER 2 FILES AND DIRECTORIES
	CREATE, COPY, REMOVE, AND MOVE FILES
		Creating Files
		Copying Files
		Copy Files with Command Substitution
		Deleting Files
		Moving Files
	THE BASENAME, DIRNAME, AND FILE COMMANDS
	THE wc COMMAND
	THE more COMMAND AND THE less COMMAND
	THE head COMMAND
	THE tail COMMAND
	FILE COMPARISON COMMANDS
	THE PARTS OF A FILENA
	WORKING WITH FILE PERMISSIONS
		The chmod Command
		The chown Command
		The chgrp Command
		The umask and ulimit Commands
	WORKING WITH DIRECTORIES
		Absolute and Relative Directories
		Absolute and Relative Path Names
		Creating Directories
		Removing Directories
		Changing Directories
		Renaming Directories
	USING QUOTE CHARACTERS
	STREAMS AND REDIRECTION COMMANDS
	METACHARACTERS AND CHARACTER CLASSES
		Digits and Characters
		Working with “^” and “\” and “!”
	FILENAMES AND METACHARACTERS
	SUMMARY
CHAPTER 3 USEFUL COMMANDS
	THE join COMMAND
	THE fold COMMAND
	THE split COMMAND
	THE sort COMMAND
	THE uniq COMMAND
	HOW TO COMPARE FILES
	THE od COMMAND
	THE tr COMMAND
	A SIMPLE USE CASE
	THE find COMMAND
	THE tee COMMAND
	FILE COMPRESSION COMMANDS
		The tar command
		The cpio Command
		The gzip and gunzip Commands
		The bunzip2 Command
		The zip Command
	COMMANDS FOR zip FILES AND bz FILES
	INTERNAL FIELD SEPARATOR (IFS)
	DATA FROM A RANGE OF COLUMNS IN A DATASET
	WORKING WITH UNEVEN ROWS IN DATASETS
	THE alias COMMAND
	SUMMARY
CHAPTER 4 CONDITIONAL LOGIC AND LOOPS
	ARITHMETIC OPERATIONS AND OPERATORS
	WORKING WITH ARRAYS
	ARRAYS AND TEXT FILES
	WORKING WITH VARIABLES
		Assigning Values to Variables
	WORKING WITH OPERATORS FOR STRINGS AND NUMBERS
	THE read COMMAND FOR USER INPUT
	THE test COMMAND FOR VARIABLES, FILES, AND DIRECTORIES
		Relational Operators
		Boolean Operators
		String Operators
		File Test Operators
	CONDITIONAL LOGIC WITH if/else STATEMENTS
	THE case/esac STATEMENT
	ARITHMETIC OPERATORS AND COMPARISONS
	WORKING WITH STRINGS IN SHELL SCRIPTS
		Working with Strings
	WORKING WITH LOOPS
		Using a for loop
	WORKING WITH NESTED LOOPS
	USING A while LOOP
	THE while, case, AND if/elif/fi STATEMENTS
	USING AN UNTIL LOOP
	USER-DEFINED FUNCTIONS
	CREATING A SIMPLE MENU FROM SHELL COMMANDS
	SUMMARY
CHAPTER 5 PROCESSING DATASETS WITH GREPAND SED
	WHAT IS THE grep COMMAND?
	METACHARACTERS AND THE grep COMMAND
	ESCAPING METACHARACTERS WITH THE grep COMMAND
	USEFUL OPTIONS FOR THE grep COMMAND
		Character Classes and the grep Command
	WORKING WITH THE –C OPTION IN grep
	MATCHING A RANGE OF LINES
	USING BACK REFERENCES IN THE grep COMMAND
	FINDING EMPTY LINES IN DATASETS
	USING KEYS TO SEARCH DATASETS
	THE BACKSLASH CHARACTER AND THE grep COMMAND
	MULTIPLE MATCHES IN THE GREP COMMAND
	THE grep COMMAND AND THE xargs COMMAND
		Searching zip Files for a String
	CHECKING FOR A UNIQUE KEY VALUE
		Redirecting Error Messages
	THE egrep COMMAND AND fgrep COMMAND
		Displaying “Pure” Words in a Dataset with egrep
		Redirecting Error Messages
	THE egrep COMMAND AND fgrep COMMAND
		Displaying “Pure” Words in a Dataset with egrep
		The fgrep Command
	DELETE ROWS WITH MISSING VALUES
	A SIMPLE USE CASE
	WHAT IS THE sed COMMAND?
		The sed Execution Cycle
	MATCHING STRING PATTERNS USING sed
	SUBSTITUTING STRING PATTERNS USING sed
		Replacing Vowels from a String or a File
		Deleting Multiple Digits and Letters from a String
	SEARCH AND REPLACE WITH sed
	DATASETS WITH MULTIPLE DELIMITERS
	USEFUL SWITCHES IN sed
	WORKING WITH DATASETS
		Printing Lines
		Character Classes and sed
		Removing Control Characters
	COUNTING WORDS IN A DATASET
	BACK REFERENCES IN sed
	ONE-LINE sed COMMANDS
	POPULATE MISSING VALUES WITH THE sed COMMAND
	A DATASET WITH 1,000,000 ROWS
		Numeric Comparisons
		Counting Adjacent Digits
		Average Support Rate
	SUMMARY
CHAPTER 6 PROCESSING DATASETS WITH AWK
	THE awk COMMAND
		Built-in Variables that Control awk
		How Does the awk Command Work?
	ALIGNING TEXT WITH THE printf COMMAND
	CONDITIONAL LOGIC AND CONTROL STATEMENTS
		The while Statement
		A for loop in awk
		A for loop with a break Statement
		The next and continue Statements
	DELETING ALTERNATE LINES IN DATASETS
	MERGING LINES IN DATASETS
		Printing File Contents as a Single Line
		Joining Groups of Lines in a Text File
		Joining Alternate Lines in a Text File
	MATCHING WITH METACHARACTERS AND CHARACTER SETS
	PRINTING LINES USING CONDITIONAL LOGIC
	SPLITTING FILENAMES WITH awk
	WORKING WITH POSTFIX ARITHMETIC OPERATORS
	NUMERIC FUNCTIONS IN awk
	ONE-LINE awk COMMANDS
	USEFUL SHORT awk SCRIPTS
	PRINTING THE WORDS IN A TEXT STRING IN awk
	COUNT OCCURRENCES OF A STRING IN SPECIFIC ROWS
	PRINTING A STRING IN A FIXED NUMBER OF COLUMNS
	PRINTING A DATASET IN A FIXED NUMBER OF COLUMNS
	ALIGNING COLUMNS IN DATASETS
	ALIGNING COLUMNS AND MULTIPLE ROWS IN DATASETS
	DISPLAYING A SUBSET OF COLUMNS IN A TEXT FILE
	SUBSETS OF COLUMN-ALIGNED ROWS IN DATASETS
	COUNTING WORD FREQUENCY IN DATASETS
	DISPLAYING ONLY “PURE” WORDS IN A DATASET
	DELETE ROWS WITH MISSING VALUES
	WORKING WITH MULTI-LINE RECORDS IN AWK
	A SIMPLE USE CASE
	ANOTHER USE CASE
	A DATASET WITH 1,000,000 ROWS
		Counting Adjacent Digits
		Average Support Rate
	SUMMARY
CHAPTER 7 PROCESSING DATASETS (PANDAS)
	PREREQUISITES FOR THIS CHAPTER
	ANALYZING MISSING DATA
		Causes of Missing Data
	PANDAS, CSV FILES, AND MISSING DATA
		Single Column CSV Files
		Two Column CSV Files
	MISSING DATA AND IMPUTATION
		Counting Missing Data Values
		Drop Redundant Columns
		Remove Duplicate Rows
		Display Duplicate Rows
		Uniformity of Data Values
		Too Many Missing Data Values
		Categorical Data
		Data Inconsistency
		Mean Value Imputation
		Random Value Imputation
		Multiple Imputation
		Matching and Hot Deck Imputation
		Is a Zero Value Valid or Invalid?
	SKEWED DATASETS
	CSV FILES WITH MULTI-ROW RECORDS
	COLUMN SUBSET AND ROW SUBRANGE OF THE TITANIC CSV FILE
	DATA NORMALIZATION
		Assigning Classes to Data
		Other Data Cleaning Tasks
		DeepChecks and Data Validation
	HANDLING CATEGORICAL DATA
		Processing Inconsistent Categorical Data
		Mapping Categorical Data to Numeric Values
		Mapping Categorical Data to One Hot Encoded Values
	WORKING WITH CURRENCY
	WORKING WITH DATES
		Find Missing Dates
		Find Unique Dates
		Switch Date Formats
	WORKING WITH IMBALANCED DATASETS
		Data Sampling Techniques
		Removing Noisy Data
		Cost-sensitive Learning
		Detecting Imbalanced Data
		Rebalancing Datasets
		Specify stratify in Data Splits
	WHAT IS SMOTE?
	DATA WRANGLING
		Data Transformation: What Does This Mean?
	A DATASET WITH 1,000,000 ROWS
		Dataset Details
		Numeric Comparisons
		Counting Adjacent Digits
	SAVING CSV DATA TO XML, JSON, AND HTML FILES
	SUMMARY
CHAPTER 8 NOSQL, SQLITE, AND PYTHON
	NON-RELATIONAL DATABASE SYSTEMS
		Advantages of Non-relational Databases
	WHAT IS NOSQL?
		What is NewSQL?
	RDBMS VERSUS NOSQL: WHICH ONE TO USE?
		Good Data Types for NoSQL
		Some Guidelines for Selecting a Database
		NoSQL Databases
	WHAT IS MONGODB?
		Features of MongoDB
		Installing MongoDB
		Launching MongoDB
	USEFUL MONGO APIS
		Metacharacters in Mongo Queries
	MONGODB COLLECTIONS AND DOCUMENTS
		Document Format in MongoDB
	CREATE A MONGODB COLLECTION
	WORKING WITH MONGODB COLLECTIONS
		Find All Android Phones
		Find All Android Phones in 2018
		Insert a New Item (Document)
		Update an Existing Item (Document)
		Calculate the Average Price for Each Brand
		Calculate the Average Price for Each Brand in 2019
		Import Data with mongoimport
	WHAT IS FUGUE?
	WHAT IS COMPASS?
	WHAT IS PYMONGO?
	MYSQL, SQLALCHEMY, AND PANDAS
		What is SQLAlchemy?
		Read MySQL Data via SQLAlchemy
	EXPORT SQL DATA FROM PANDAS TO EXCEL
	MYSQL AND CONNECTOR/PYTHON
		Establishing a Database Connection
		Creating a Database Table
		Reading Data from a Database Table
	WHAT IS SQLITE?
		SQLite Features
		SQLite Installation
		SQLiteStudio Installation
		DB Browser for SQLite Installation
		SQLiteDict (Optional)
	WHAT IS TIMESCALEDB?
		Install Timescaledb (Macbook)
		Setting Up the TimescaleDB Extension
		The rides Table
		The Parallel Copy Command
		Data Analysis
	LARGE SCALE DATA IMPUTATION
	SUMMARY
INDEX