Scripting: Automation with Bash, PowerShell, and Python

Cover
Contents
	Preface
		Bash, PowerShell, or Python?
		About This Book
			Scripting and AI Tools
			Scripting as a Core Competence for Efficient IT Work
	Part I: Scripting Languages
		1  Scripting: Doing One Thing
			1.1 What Does Scripting Mean?
				1.1.1 Scripting versus Programming
				1.1.2 Glue Languages and Glue Code
				1.1.3 Do One Thing and Do It Well
			1.2 Scripting Languages
				1.2.1 Bash and Zsh
				1.2.2 PowerShell
				1.2.3 Python
				1.2.4 Many Similarities, Even More Differences
			1.3 The Agony of Choice
		2  Ten Times Ten Lines
			2.1 Markdown Spell Checker (Bash)
			2.2 Sorting Images by Date (PowerShell)
			2.3 Converting a JSON File to XML Format (Python)
			2.4 Daily Server Backups (Bash)
			2.5 Web Scraping (Python)
			2.6 Logging Weather Data (Python)
			2.7 Microsoft Hyper-V Cleanup (PowerShell)
			2.8 Statistical Analysis of a Logging File (Bash)
			2.9 Uploading Files to the Cloud (PowerShell)
			2.10 Cloning Virtual Machines (Bash)
		3  Bash and Zsh
			3.1 Terminal, Shell, and Bash
				3.1.1 Determining the Current Shell
			3.2 Installation
				3.2.1 Using Bash in a Virtual Machine
				3.2.2 Windows Subsystem for Linux
				3.2.3 Git Bash
				3.2.4 Docker
				3.2.5 Bash Configuration (/etc/profile and .bashrc)
			3.3 Running Commands Interactively
				3.3.1 Operating the Terminal
				3.3.2 Online Help
				3.3.3 Command Reference
			3.4 Zsh as an Alternative to Bash
				3.4.1 Installation
				3.4.2 Zsh Configuration (/etc/zshrc and .zshrc)
				3.4.3 Oh My Zsh
			3.5 The First Bash Script
				3.5.1 Hash Bang (Shebang)
				3.5.2 Shebang Variants
				3.5.3 Making the Script Executable (chmod +x)
				3.5.4 Hello, World!
				3.5.5 Elementary Syntax Rules
				3.5.6 A “Real” Example: Backup Script
				3.5.7 Running Scripts without ./
			3.6 Running Commands
				3.6.1 Serial Execution
				3.6.2 Conditional Command Execution
				3.6.3 Running Commands in the Background
				3.6.4 Running Commands in a Subshell
			3.7 Standard Input and Standard Output
				3.7.1 Redirecting Input and Output
				3.7.2 The Pipe Operator
				3.7.3 Redirecting and Displaying Outputs at the Same Time
			3.8 Globbing, Brace Extension, and Handling File and Directory Names
				3.8.1 Recursive Globbing
				3.8.2 Access to Important Directories
				3.8.3 Brace Extension
				3.8.4 Filenames and Directory Names with Spaces
			3.9 Variables
				3.9.1 Initializing and Deleting Variables
				3.9.2 Declaring Variables
				3.9.3 Performing Calculations in Bash
				3.9.4 Arrays
				3.9.5 Predefined Variables
				3.9.6 Environment Variables
			3.10 Strings
				3.10.1 Single versus Double Quotation Marks
				3.10.2 Outputting Strings (echo)
				3.10.3 Colors
				3.10.4 Entering or Reading Character Strings (read)
				3.10.5 Substitution and Expansion Mechanisms
				3.10.6 Parameter Substitution
				3.10.7 Heredocs and Herestrings
				3.10.8 The Backslash
			3.11 Branches
				3.11.1 if Short Notation with && or ||
				3.11.2 Conditions
				3.11.3 Branches with case
				3.11.4 Parameter Analysis via case
			3.12 Loops
				3.12.1 Processing Filenames with Spaces
				3.12.2 while and until
				3.12.3 Loops over Text Files
			3.13 Functions
				3.13.1 Local Variables
			3.14 Error Protection
				3.14.1 Detecting Errors
				3.14.2 Canceling in Case of Errors
				3.14.3 exit
				3.14.4 Responding to Signals (trap)
				3.14.5 Timeout
		4  PowerShell
			4.1 Installation
				4.1.1 Installation on Linux
				4.1.2 Installation on macOS
				4.1.3 Limitations of PowerShell on Windows versus Linux/macOS
				4.1.4 Configuration
			4.2 Windows Terminal
				4.2.1 Configuration
				4.2.2 Operation
			4.3 Calling cmdlets and Functions
				4.3.1 cmdlets
				4.3.2 The Verb-Noun Terminology
				4.3.3 Aliases
				4.3.4 Parameters and Options
				4.3.5 Splatting
				4.3.6 Functions
				4.3.7 Running Conventional Commands
				4.3.8 Online Help
			4.4 Combining Commands
				4.4.1 Chains of Commands
				4.4.2 Multiline Statements
				4.4.3 Dealing with Special Characters
			4.5 The First Script
				4.5.1 Hello, World!
				4.5.2 Trouble with the Execution Policy
				4.5.3 Setting Up a Custom Script Directory
				4.5.4 Running Scripts on Linux and macOS
				4.5.5 Example: Cleaning Up the Downloads Directory
				4.5.6 Differentiating Between PowerShell Versions
			4.6 Variables, Strings, and Objects
				4.6.1 Data Types
				4.6.2 Calculating and Comparing
				4.6.3 Strings
				4.6.4 Command Substitution
				4.6.5 Objects
				4.6.6 Assembling Custom Objects
				4.6.7 Predefined Variables
				4.6.8 Environment Variables
			4.7 Arrays and Hash Tables
				4.7.1 Creating and Processing Custom Arrays
				4.7.2 Hash Tables
			4.8 Output Redirection
				4.8.1 Streams
				4.8.2 Output Redirection
				4.8.3 Duplicating Output (Tee-Object)
				4.8.4 (Not) Formatting the Output
				4.8.5 No Input Redirection
			4.9 Loops
				4.9.1 ForEach-Object
				4.9.2 for, while, do-while, and do-until
			4.10 Branches
				4.10.1 Case Distinctions Using switch
			4.11 Functions and Parameters
				4.11.1 Function Results
				4.11.2 return
				4.11.3 Parameters
				4.11.4 Calling Functions
				4.11.5 Processing Standard Input
				4.11.6 Syntax Variants
			4.12 Modules
				4.12.1 Module Directories
				4.12.2 Modules for Experts
			4.13 Error Protection
				4.13.1 Set-StrictMode
				4.13.2 Aborting Scripts on Errors
				4.13.3 Separating Error Messages from Other Output
				4.13.4 Error Protection Using try/catch
				4.13.5 Terminating the Script Execution Prematurely (Exit)
		5  Python
			5.1 Installing Python
				5.1.1 Windows
				5.1.2 macOS
			5.2 Getting to Know Python in a Terminal Window
			5.3 Programming Custom Scripts
				5.3.1 Shebang
			5.4 Elementary Syntax Rules
				5.4.1 Block Elements
				5.4.2 Comments
			5.5 Numbers
				5.5.1 Floats
				5.5.2 Random Numbers
			5.6 Strings
				5.6.1 Raw Strings
				5.6.2 Processing Strings
				5.6.3 Slicing
				5.6.4 print and input
				5.6.5 Formatting Outputs
			5.7 Lists
				5.7.1 List Comprehension
				5.7.2 map and filter
			5.8 Tuples, Sets, and Dictionaries
				5.8.1 Sets
				5.8.2 Dictionaries
			5.9 Variables
				5.9.1 Assignments
				5.9.2 Data Types
				5.9.3 Mutable or Immutable?
			5.10 Operators
				5.10.1 Combining Assignment and Evaluation
			5.11 Branches (if)
				5.11.1 Conditions
				5.11.2 Short Notation with if
			5.12 Loops (for and while)
				5.12.1 break, continue, and else
				5.12.2 Loops over Number Ranges (range)
				5.12.3 Loops over the Characters of a String
				5.12.4 Loops over Lists, Tuples, and Sets
				5.12.5 Loops over Dictionaries
				5.12.6 Loops over the Parameters of a Script
				5.12.7 Globbing on Windows
				5.12.8 Loops over the Lines of a Text File
				5.12.9 Loops over All Files in a Directory
			5.13 Functions
				5.13.1 Global and Local Variables
				5.13.2 Parameters
				5.13.3 Optional Parameters
				5.13.4 Variable Number of Parameters
				5.13.5 Lambda Functions
			5.14 Processing Text Files
				5.14.1 Example: Analyzing a CSV File
				5.14.2 Example: Creating SQL Commands
			5.15 Error Protection
				5.15.1 try/except
			5.16 System Functions
				5.16.1 Access to the Program Parameters
				5.16.2 Access to Standard Input and Standard Output
				5.16.3 Exiting a Program
				5.16.4 Calling Linux Commands
				5.16.5 Processing Results
				5.16.6 Running Commands through the Shell
				5.16.7 Error during a Command Call
				5.16.8 Waiting (sleep)
			5.17 Modules
				5.17.1 import
				5.17.2 Distributing Custom Scripts across Multiple Files
			5.18 Installing Additional Modules Using pip
				5.18.1 Installing pip
				5.18.2 Applying pip
				5.18.3 pip Problems with Windows
				5.18.4 pip Problems with Linux
				5.18.5 requirements.txt
				5.18.6 pipenv
	Part II: Work Techniques and Tools
		6  Linux Toolbox
			6.1 Directories and Files
				6.1.1 Listing, Copying, Moving, and Deleting Files
			6.2 Finding Files
				6.2.1 Text Search Using grep
			6.3 Compressing and Archiving Files
				6.3.1 Archiving Files Using tar
				6.3.2 ZIP Files
			6.4 Using Root Privileges
				6.4.1 sudo Privileges for Selected Accounts Only
				6.4.2 Issues with Output Redirection
				6.4.3 Access Rights
				6.4.4 Managing Processes
				6.4.5 Terminating Processes
				6.4.6 Background Processes and System Services
				6.4.7 Logging Files
				6.4.8 Determining Free Memory
				6.4.9 Determining Other System Information
			6.5 Software Installation
				6.5.1 Updating Software
				6.5.2 Installing Additional Packages
			6.6 Other Commands
		7  cmdlets for PowerShell
			7.1 Directories and Files
				7.1.1 Listing, Copying, Moving, and Deleting Files
				7.1.2 Reading and Writing Text
				7.1.3 Inputting and Outputting Text
			7.2 Finding Files
				7.2.1 Searching Text Files
				7.2.2 Text versus Objects
			7.3 Compressing and Archiving Files
				7.3.1 Compressing Search Results
			7.4 Process Management
				7.4.1 Launching Other PowerShell Scripts
				7.4.2 Running Commands in the Background
				7.4.3 Managing Services
				7.4.4 Working on Other Computers
			7.5 Registration Database and System Information
				7.5.1 Determining System Information
				7.5.2 Evaluating the Logging System
			7.6 Processing cmdlet Results
				7.6.1 Select-Object and Sort-Object
				7.6.2 Where-Object
				7.6.3 Group-Object
				7.6.4 ForEach-Item
				7.6.5 Measure-Object
				7.6.6 Formatting and Exporting cmdlet Results
			7.7 Other cmdlets
			7.8 Installing Additional Modules
				7.8.1 Listing Modules, Commands, and Package Sources
				7.8.2 NuGet and winget
			7.9 Standard Aliases
		8  Analyzing Texts with Filters and Pipes
			8.1 grep, sort, cut, and uniq
				8.1.1 grep
				8.1.2 wc
				8.1.3 cut
				8.1.4 sort
				8.1.5 head and tail
				8.1.6 uniq
				8.1.7 tr
				8.1.8 awk and sed
			8.2 Example: Statistical Data Analysis
				8.2.1 Data Extraction via Script
			8.3 Example: ping Analysis
				8.3.1 ping Call via Script
			8.4 Example: Apache Log Analysis
				8.4.1 Extracting IP Addresses
				8.4.2 Identifying Popular Pages
			8.5 CSV Files
				8.5.1 Processing CSV Files Using Python
				8.5.2 Processing CSV Files in PowerShell
		9  Regular Expressions
			9.1 Syntax Rules for Regular Expressions
				9.1.1 Characters
			9.2 Groups and Alternatives
				9.2.1 Quantifiers
				9.2.2 On Greed (Greedy versus Lazy)
				9.2.3 Alpha and Omega
				9.2.4 Exercise Pages and Cheat Sheets
				9.2.5 Example: Recognizing a Date
				9.2.6 Example: Recognizing IPv4 Addresses
				9.2.7 Example: Recognizing Email Addresses
			9.3 Regular Expressions in Bash (grep, sed)
				9.3.1 Bash Comparison Operator for Regular Expressions
				9.3.2 Filtering Text Using grep
				9.3.3 The \"sed\" Stream Editor
				9.3.4 Searching and Replacing Using sed
				9.3.5 sed Example: Manipulating Paths of Image Files
			9.4 Regular Expressions in PowerShell
				9.4.1 The match and replace Operators
				9.4.2 Select-String
				9.4.3 Example: Input Validation
				9.4.4 Example: List of Figures
			9.5 Regular Expressions in Python (re Module)
				9.5.1 Example: Verifying a MAC Address
				9.5.2 Example: Anonymizing a Logging File
		10  JSON, XML, and INI
			10.1 JSON in PowerShell
				10.1.1 Example: Saving Event Log Entries in JSON Format
				10.1.2 Example: Analyzing Domain Queries
				10.1.3 Example: Converting between CSV and JSON
			10.2 JSON and Python
				10.2.1 Example: Collecting Birthdays
				10.2.2 Example: Determining Holidays
			10.3 JSON in Bash
				10.3.1 Viewing JSON Files Interactively Using fx
			10.4 XML in PowerShell
				10.4.1 XML Data Type
				10.4.2 Select-Xml
				10.4.3 ConvertTo-Xml, Export-Clixml, and Import-Clixml
			10.5 XML and Python
				10.5.1 Example: Creating a Dictionary for Country Codes
				10.5.2 Example: Analyzing an RSS Feed
				10.5.3 Example: Extracting MAC Addresses from Virtual Machine Files
			10.6 XML in Bash
				10.6.1 xmllint
				10.6.2 XMLStarlet
			10.7 INI Files
				10.7.1 Python
				10.7.2 PowerShell
				10.7.3 Bash
		11  Running Scripts Automatically
			11.1 cron
				11.1.1 /etc/crontab
				11.1.2 Personal crontab Files
				11.1.3 Hourly, Daily, Weekly, and Monthly Directories
				11.1.4 Alternatives to cron
				11.1.5 Starting Jobs Automatically on macOS
			11.2 Example: Web Server Monitoring
				11.2.1 Testing and Troubleshooting
				11.2.2 Real Monitoring
			11.3 Microsoft Windows Task Scheduler
				11.3.1 Troubleshooting
				11.3.2 Setting Up Tasks via cmdlets
			11.4 Example: Saving Exchange Rates
			11.5 Tracking File System Changes
				11.5.1 inotify
				11.5.2 Alternatives to inotify
		12  SSH
			12.1 Installing the SSH Client and Server
				12.1.1 Linux
				12.1.2 macOS
				12.1.3 Windows
				12.1.4 Using an Editor in an SSH Session
				12.1.5 Securing an SSH Server
			12.2 Working with SSH
				12.2.1 Host Verification
				12.2.2 Potential Issues and Their Cause
				12.2.3 Running Linux and macOS Commands
				12.2.4 Running Windows Commands
				12.2.5 SSH Remoting in PowerShell
			12.3 scp and rsync
				12.3.1 Copy-Item with SSH Remoting
				12.3.2 rsync
			12.4 SSH Authentication with Keys
				12.4.1 Generating a Key Pair
				12.4.2 Storing the Public Component of the Key on the Server (macOS and Linux)
				12.4.3 Storing the Public Component of the Key on the Server (Windows)
			12.5 Example: Uploading Images to a Linux Web Server
				12.5.1 Preparatory Measures
				12.5.2 Bash Script
				12.5.3 PowerShell Script
			12.6 Example: Analyzing Virtual Machines
		13  Visual Studio Code
			13.1 Introduction
				13.1.1 Comparing VS Code, VSCodium, and Visual Studio
				13.1.2 Think in Terms of Directories, Not Files!
			13.2 Language-Specific VS Code Extensions
				13.2.1 PowerShell Extension
				13.2.2 Python Extension
				13.2.3 Bash/Shell Extensions
			13.3 Remote – SSH Extension
				13.3.1 Applying the Remote – SSH Extension
				13.3.2 Limitations
		14  Git
			14.1 Git Crash Course
				14.1.1 Preparation Tasks
				14.1.2 The First Repository
				14.1.3 The First Commit
				14.1.4 Additional Commits
				14.1.5 Setting Up the Repository on a Second Computer
				14.1.6 Git Status
				14.1.7 Excluding Files from Git (.gitignore)
				14.1.8 Transferring Existing Code to a New Repository
			14.2 Handling Settings and Passwords Correctly
				14.2.1 What’s a Better Method?
				14.2.2 Example
			14.3 Git Automation
				14.3.1 Controlling GitHub Remotely Using gh
			14.4 Git Hooks
				14.4.1 Example: Detecting Unversioned Files Before the Commit
	Part III: Applications and Examples
		15  Backups
			15.1 Synchronizing Directories to External Storage
				15.1.1 PowerShell Script with robocopy
				15.1.2 Ideas for Improvement
				15.1.3 Bash Script with rsync
			15.2 WordPress Backup
			15.3 SQL Server Backup
				15.3.1 Backing Up and Compressing all Databases via Script
		16  Image Processing
			16.1 Manipulating Image Files
				16.1.1 Installing ImageMagick
				16.1.2 Trying Out ImageMagick
				16.1.3 Example: convert2eps (Bash Variant)
				16.1.4 Example: convert2eps (PowerShell Variant)
			16.2 Sorting Photos by Date Taken
				16.2.1 Installing and Trying Out ExifTool
				16.2.2 Bash Example
				16.2.3 PowerShell Example
			16.3 Converting Exif Metadata to SQL Commands
				16.3.1 PyExifTool
				16.3.2 EXIF2SQL
				16.3.3 Options for Enhancement
		17  Web Scraping
			17.1 Limitations
			17.2 Web Scraping, Web Crawling, and Data Mining
			17.3 Downloading Websites Using wget
				17.3.1 Example 1: Downloading Directly Linked Images
				17.3.2 Example 2: Downloading all PDF Files Linked on the Website
				17.3.3 Example 3: Creating a Local Copy of the Website
			17.4 Web Scraping with Regular Expressions
			17.5 Web Scraping with Beautiful Soup
				17.5.1 Beautiful Soup and Requests
				17.5.2 Hello, Beautiful Soup!
				17.5.3 Example: Determining a List of Top Titles from Rheinwerk Publishing
			17.6 Web Scraping with Requests-HTML
				17.6.1 Example: Analyzing the GitHub Status
			17.7 Web Scraping with PowerShell
				17.7.1 PowerHTML Module
		18  Using REST APIs
			18.1 Tools
			18.2 Sample APIs to Try Out
			18.3 Implementing Custom REST APIs
			18.4 curl and wget
				18.4.1 curl
				18.4.2 wget
				18.4.3 Using REST APIs in Bash Scripts
			18.5 Using REST APIs in PowerShell
				18.5.1 Options
			18.6 Example: Determining the Current Weather
			18.7 Using REST APIs in Python
			18.8 Example: Determining Electricity Prices and Displaying Them Graphically
				18.8.1 aWATTar API
				18.8.2 Analysis of the Data
				18.8.3 Matplotlib
				18.8.4 Controlling the Energy Consumption
		19  Databases
			19.1 Updating and Maintaining Databases
				19.1.1 PowerShell and sqlcmd
			19.2 Creating a New Customer Account
				19.2.1 Account Data
				19.2.2 Structure of the Script
			19.3 Storing Exif Metadata in a Database
				19.3.1 PyMySQL
				19.3.2 Saving Exif Metadata
			19.4 Importing JSON Data into a Table
		20  Scripting in the Cloud
			20.1 AWS CLI
				20.1.1 Installation on Linux and macOS
				20.1.2 Configuration
				20.1.3 Getting Started
				20.1.4 Encrypting Files
			20.2 Example: Uploading Encrypted Backup Files to the Cloud
			20.3 AWS PowerShell Module
				20.3.1 Getting Started
				20.3.2 Copying Files
			20.4 Example: Offloading Large Files from a Website to the Cloud
				20.4.1 Preparations
				20.4.2 Script
				20.4.3 Limitations
		21  Virtual Machines
			21.1 Setting Up and Running Virtual Machines (KVMs)
				21.1.1 Cloning Virtual Machines
				21.1.2 Starting and Shutting Down Virtual Machines
				21.1.3 Running Scripts on Multiple Virtual Machines
			21.2 Automating the Network Configuration (KVMs)
				21.2.1 Starting Point
			21.3 Controlling Hyper-V
				21.3.1 Cloning Virtual Machines
	The Author
	Index