Linux System Administration For The 2020s: The Modern Sysadmin Leaving Behind The Culture Of Build And Maintain

Table of Contents
About the Author
About the Technical Reviewer
Acknowledgments
Introduction
Part I: Laying the Foundation
	Chapter 1: Linux at a Glance
		Brief Unix to Linux History
		Open Source
		Linux Is Everywhere
		Community Linux Distributions
			Community
			Upstream
			Community Contributors
			Common Distributions
		Which Distribution Is Best for You
			Before Committing
			The Three Linux Distro Categories
				Option One: Out-of-the-Box Distros
					Easy to Understand
					Installation Should Not Require a Degree
					Try Ubuntu
						Walk Before Running
				Option Two: The Almost Out-of-the-Box Distros
					Try Fedora, openSUSE, or Debian
				Option Three: The “Challenge Accepted” Distros
					With Great Power …
					Try Arch Linux or Gentoo
		Enterprise Linux Distributions
			Red Hat
				Red Hat Enterprise Linux
				Automation
				Hybrid Cloud
			Canonical
				Linux Support
				Cloud
				Internet of Things
			SUSE
				Server and Desktop
				Cloud, Storage, and Management
		Community vs. Enterprise
		Knowledge Check
		Summary
Part II: Strengthening Core Skills
	Chapter 2: New Tools to Improve the Administrative Experience
		Task Management
			Starting a Process
			Task Visualization Tooling
				Top
				Alternatives to Top
				nmon
			Killing Processes
			Zombie Processes
			Background Tasks
			Running Time-Consuming Tasks
				Screen
				Tmux
		Ansible Introduction
			Installing Ansible
				Package Management
				Pip
			Configuring Ansible
			Ansible Inventory
			Running Ansible
			Playbooks
			Roles
				Role Directory Structure
				Generating Ansible Roles
			Modules
			Sharing Your Ansible
				Ansible Galaxy
		Web Consoles
			Cockpit
				Installation
				Configuration
				Using Cockpit
				Limitations
			Alternatives to Cockpit
				Webmin
				Ajenti
		Text Consoles
			Installing
			Using
		Summary
	Chapter 3: Estate Management
		Outdated Ways of Working
			Outdated Skills
			Keeping Knowledge to Themselves
			Over Engineering
			Shell Scripting
			Snowflakes
			Reinventing the Wheel
		Build Process
			Manual Installation Methods
				Boot Media Install
				Network Install
				Templates
				Virtual Machine Images
			Automated Linux Installations
				Method 1: Network Install
					PXE Server
					Kickstart
				Method 2: Virtual Machine Templates
					Hypervisor API
				Ansible Examples
			Using Images
				Golden Image
				Use It
				Don’t Use It
				Image Catalog
					Advantages
					Disadvantages
			Build Process Flow
				Basic Build Process
				What Can Be Improved
					Automate, Automate, Automate
					Introduce a User Request Portal
						Integration with Other Platforms
						Simplify Resource Requirements
					Use an Automation Platform
					Introduce Expiry Dates
				Automated Build Process Flow
		System Patching
			Update Types
				Package Updates
				Errata
			Staging
			Patch Management Systems
			Planning
			Rollback
				System Restore from Backup
				Restore Snapshot
				Package Management Rollback
				Reinstallation of Packages
				Redeployment of System
		Backup and Recovery
			Important Directories and Files
			Virtual Machine Backups
			Disaster Recovery
				Best Strategies Based on Recovery Times
					Replicated Data Centers
					Stretched Clusters
					Infrastructure As Code
					Cloud
		Common Bad Practices
			Virtual Machine Templates
			Patching or Lack Thereof
			Firewall Disabled
			SELinux Disabled or Permissive
			Using Community Repositories
			Scripts, Scripts, and More Scripts
			Running As Root
		Good Practices
			Building Throwaway Systems
			Automate As Much As Possible
			Search Before Creating
			Sharing Knowledge and Collaborating
			Source Control
			Reassessing System Requirements
		Summary
	Chapter 4: Estate Management Tools
		Management Systems
		Linux Platform Tools
		Linux Platform Tools Available
			Selecting Your Linux Platform Tool
			The Decision
			Satellite Server
				Satellite 5
				Configuration Management
				System Deployment
			Satellite 6
				Content Management
				Content Views
				Life Cycles
				Content Management Flow
				System Provisioning
				System Patching
				Configuration Management
				Reasons to Use Satellite
				Reasons to Not Use Satellite
			SUSE Manager
				Uyuni
				Support
				SUSE Manager Configuration
				Reasons to Use SUSE Manager
				Reasons to Not Use SUSE Manager
			Foreman
				Provision Hypervisors
				Plugins
				Open Source Does Need Money Too
			Spacewalk
				Abandoned
				Why It Was Good
				Network Provisioning
				Environment Staging
				Thank You for Your Service
			Provisioning Tools
			Cloudforms
				Single Pane of Glass
				State Machines
				User Request Portal
				Chargeback
				Request Approvals
				Advantages
				Disadvantages
			Terraform
				Products Available
				Community CLI
				Terraform Cloud Platform
		API and Extracting Useful Information
			Don’t Reinvent the Wheel
			Why to Not Write Your Own Tool
		Best Tools to Use
			Pipeline Tooling
			Automation Platforms
			Shell Scripts
		Summary
	Chapter 5: Automation
		Automation in Theory
			Idempotent Code
			Knowing When and When Not to Automate
				Reasons to Automate
				Reasons Not to Automate
			State Management
		Automation Tooling
			Automation Scripting Languages
				YAML
				These Are Not the Spaces You Are Looking For
				YAML in Action
					Ansible
					SaltStack
				Ruby
				Python
				Shell Scripting
		Automation Platforms
			Automation in Estate Management Tools
				Reasons to Use
				Reasons Not to Use
			Ansible Automation Platform
				Agentless
				Potential Security Hole
				Using Ansible
					Command Line
				Graphical User Interface
					Reasons to Use Ansible
					Reasons Not to Use Ansible
				AWX
					Reasons to Use AWX
					Reasons Not to Use AWX
				SaltStack
				Server to Client Communication
				Remote Execution
				Configuration Management
				Uses a Message Bus
					Reasons to Use SaltStack
					Reasons to Not Use SaltStack
				Puppet
					Red Hat and Puppet
					Server Agent Based
					Potential Lower Adoption
					Enterprise and Community
					Reasons to Use Puppet
					Reasons to Not Use Puppet
				Chef
					Ways to Use Chef
						Managed Service
						On-Premise
						Community
					Reasons to Use Chef
					Reasons to Not Use Chef
			Making the Decision
				Market Trends
				See for Yourself
				Enterprise vs. Community vs. Cost
				Product Life Cycle
		Automation with Management Tools
			State Management
			Enterprise Products
			Use Case Example
				The Platform Tool
				The Platform Tool Configuration
				The Mistake
				Laying in the Shadows Waiting
				Safety Net
			Setting Up a SOE
				Build from a Standard
				Source Control
				Phased Testing
					Code Development
					Code Testing and Peer Reviewed
					Code Promotion
		Automate the Automation
			Self-Healing
				Self-Healing Layers
					Removing All Single Points of Failure
					Hardware Layer Self-Healing
						Reporting
						Ensuring Platform Availability
						Automated Recovery
					Platform Layer Self-Healing
					Application Layer Self-Healing
			When to Self-Heal
			How to Implement Self-Healing
				Gates
				Tooling: Automation and State Management
				Machine Learning
				Off-the-Shelf Products
					Dynatrace
		Automation Best Practices
			Do Not Reinvent the Wheel, Again …
				Code Libraries
					Ansible
					Puppet
					SaltStack
				Metadata
			Things to Avoid
			Shell Scripts
			Restarting Services When Not Required
			Using Old Versions
			Correct Version Documentation
			Good Practices
				Debugging
				Don't Forget README
				Source Control
		Summary
	Chapter 6: Containers
		Getting Started
			Virtual Machine vs. Container
			Container History
			Container Runtimes
				Low-Level or OCI Runtimes
					Native Runtimes
					Virtual and Sandboxed Runtimes
						Sandbox Runtimes
						Virtual Runtimes
				Container Runtime Interface
					Containerd
					CRI-O
				Container Engines
					Docker
					Podman
			Container Images
				Container Registries
					Cloud Registries
					Local Registries
				Container Registry Providers
		Containers in Practice
			Prerequisites
				Shopping List
				System Prep
					Install Packages
			Creating Containers
				Pulling a Container Image
					Finding Container Images
					Pulling the Container Image
					Local Container Images
				Running a Container
				Running Containers
			Custom Images and Containers
				Create a Podman Image Registry
					Create a Directory for Data to Be Stored
					Create Registry Container
					Set Podman to Use Insecure Registry
				Using the Podman Registry
					Tagging Images
					Pushing Images
					Remote Registries
				Customize an Image
					Dockerfile
					Example
						Pull Down CentOS Image
						Dockerfile
						Build Image
						Create Container
						Confirm Container Is Running
						Delete Container
		Container Practices
			Cloud Native
			Good Practices
				Keep It Small
				Dynamic Deployment
				Scalable
				“Does It Cloud”?
			Bad Practices
				Containers Are Not Virtual Machines
				Different Images
				Production Builds from Code
				Hardcoded Secrets or Configuration
				Building Idempotent Containers
		Container Development
			Development Considerations
				Coding Languages
				Code Editor
				Source Control
			Container Tooling
				CI/CD
					Jenkins Example
				Dedicated Image Builders
				Image Registry
				Development Editor Plugins
				Linting Tools
		DevSecOps
			DevSecOps Tooling
				Pipelines
				Security Gates
		GitOps
			GitOps Toolbox
				Git
				Infrastructure As Code
				Pipeline Tools
				ArgoCD
		Container Orchestration
			What Does It Do?
			Why Not Use Podman?
			Orchestration Options
				Kubernetes
					Kubernetes Forks
					Master Components
						The Control Plane
						Nodes
						Namespaces
						Daemonsets
					Worker Node Components
						Pods
						Services
						Volumes
						Configmaps
				OpenShift
					Early OpenShift
					Current OpenShift
					OpenShift Components
						Product
						Enterprise
						Security
						Web Console
					Many More
		Summary
Part III: Day Two Practices and Keeping the Lights On
	Chapter 7: Monitoring
		Linux Monitoring Tools
			Process Monitoring
				Default Process Commands, ps and top
				Pstree
				Resource-Hungry Processes
					Memory-Intensive Processes
					CPU-Intensive Processes
			Disk and IO
				iostat and iotop
				du and df
			CPU
				Top
				mpstat
			Memory
				Free
				Page Size
				Huge Page Size
				pmap
			Virtual Memory
				vmstat
			Network
				Netstat
				ss
				iptraf-ng
				Tcpdump
				NetHogs
				iftop
			Graphical Tools
				Gnome System Monitor
				Ksysguard
			Historical Monitoring Data
				Sar
				Performance Co-Pilot
				vnstat
		Central Monitoring
			Nagios
				Versions
					Core
					Nagios XI
				Agent Based
					NRPE
					NRDP
					NSClient++
					NCPA
				Nagios Forks
				Installation
			Prometheus
				Exporters
				Alert Tool
				Dashboarding
				Query Language
				Installation
					Kubernetes or OpenShift
				Configuration
					Global
					Rule_files
					Scrape_configs
				Starting Prometheus
			Thanos
				Sidecar
				Store Gateway
				Compactor
				Receiver
				Ruler/Rule
				Querier
				Query Frontend
				Thanos Basic Layout
			Enterprise Monitoring
				Zabbix
					Enterprise Support
					Installation
					Useful Features
				CheckMk
					Enterprise Support
					Installation
					Useful Features
				OpenNMS
					Enterprise Support
					Installation
					Useful Features
		Dashboards
			Dashboarding Tools
			Grafana
				What Is Grafana
				Using Grafana
				Cloud Service
				On-Premise Installation
				Data Sources
				Dashboard Creation
					Panels
					Rows
					Save
		Application Monitoring
			Tracing Tools
				Jaeger
				Zipkin
			Exposing Metrics
				How to Speak “Developer”
		Summary
	Chapter 8: Logging
		Linux Logging Systems
			Rsyslog
				Modular
				Installation
				Service
				Configuration Files
					Global Directives
					Templates
					Rules
						Selector Field
					Action Field
		Fluentd
			Plugin Based
			Used at Scale
			Installation
				Prerequisites
				Manual Installation
				Container Deployment
			Configuration
		Understanding Logs
			Where Are the Log Files
			How to Read Log Files
			Infrastructure Logs
				Important Logs
					/var/log/messages
					/var/log/secure
					/var/log/boot.log
					/var/log/dmesg
					/var/log/yum.log
					/var/log/cron
			Application Logs
				Good Practice
					Use /var/log Directory for Logs
					Security
					Warn or Above
			Increasing Verbosity
				Log Verbosity Levels
		Log Maintenance
			Log Management Tools
				Logrotate
					Installation
		Log Forwarding
			Central Logging Systems
				Elastic Stack
				Fluentd
					Log Forwarders
					Log Aggregators
				Rsyslog
					Rsyslog Aggregator
					Rsyslog Forwarders
		Summary
	Chapter 9: Security
		Linux Security
			Standard Linux Security Tools
				Firewall
				Iptables
				Firewalld
				SELinux
				Host-Based Intrusion Detection
			Recommended Linux Security Configurations
				Disable Root Login
				Minimal Install
				Disk Partitions
				Disk Encryption
				No Desktop
				Encrypt Network Communications
				Remove and Disable Insecure or Unused Services
				Apply Updates and Patch Kernel
				SELinux and Firewall
				Improved Authentication Configuration
				Check for Open Ports
				World Writable Files
				Files Not Owned by Anyone
				ACLs
				Send Logs to Central Logging Service
				Intrusion Detection
				Application Server Security
		DevSecOps
			What Is It?
			Everyone Is Responsible for Security
			Tools
				Security Gates
				Third-Party Tools
		System Compliance
			System Hardening
				Hardening Standards
				Center for Internet Security
				Security Technical Implementation Guides
				Hardening Linux
				Manual Configuration
				Automation
				OpenSCAP
		Vulnerability Scanning
			Linux Scanning Tools
				OpenVAS
				OpenSCAP
				ClamAV
			Container Image Scanning Tools
				Harbor
				Role-Based Access
				Trivy
				Single or Multiple Images
				JFrog Xray
				Deep Scanning
				Clair
				Supported Images
				Enterprise Version
					Continuous Scanning
					Dashboard
					Pipeline
			Container Platform Scanning Tools
				Red Hat Advanced Cluster Security for Kubernetes (StackRox)
				Vulnerability Scanning
				Compliance Scanning
				Network Segmentation
				Risk Profiling
				Configuration Management
				Detection and Response
				Falco
				Flexibly Rules Engine
				Immediate Alerting
				Current Detection Rules
				Aqua Security
				Developer Guidance
				Informative Dashboarding
		Summary
	Chapter 10: Maintenance Tasks and Planning
		What Maintenance Should Be Done
			Patching
				Staging
				Sandbox
				Automated Testing
				Automated Patching
					Rollback
			Filesystem
				Cleanup
				Check for Errors
				Filesystem Check Commands
			Firewall
			Backups
			As Often As Possible
				No Live Patching Without Testing
				Structure
		How Should Maintenance Be Done
			Automation
			Zero Downtime Environments
				Blue/Green
				Failover
		Maintenance Planning
			Agree Maintenance Window
			Bite-Size Chunks
				Art of Estimating
		Automating Process and Task Together
			Process Automation
				Red Hat PAM
		Summary
Part IV: See, Analyze, Then Act
	Chapter 11: Troubleshooting
		See, Analyze, Then Act
			Understand the Problem
				Know Where to Start
					Standard Questions to Ask When Starting
				Explain the Problem
					Explain to Yourself
					Rubber Duck
					Another Person
					Use Tools
				Break Down the Problem
					Onions, They Have Layers
				The Five Whys
					Example
			Theorize Based on Evidence
				Hypothesis Building
				Build Your Theory
					Causality
				Prove Your Theory
					Reproduce the Issue
					Fix in the Test Environment
				Remediation
		Ask for Help
			What to Do Before Asking for Help
				Training
			How to Ask for Help
				Proper Grammar
				Spelling
				How to Phrase Your Questions
					A Better Question
				Where to Ask Questions
					Correct Area
					Forums
					GitHub, Stack Overflow
					Support Cases
		Things to Avoid When Troubleshooting
			Live Debugging
			Correlation vs. Causation
			Being a Lone Wolf
			Guessing and Lying
			Ghosts
			All the Small Things
			Keep Track of What You Have Tried
			Measure Twice, Cut Once
			Do Not Forget Your Retrospective
		Summary
	Chapter 12: Advanced Administration
		System Analysis
			Tools for the Sysadmin
				Sosreport
				xsos
				System Information
					Shortcut Tools
					More Details
		System Tracing
			Strace
				Installation
				Output to a File
				What to Look For
				Useful Strace Parameters
			Systemtap
				Installation
					Manual Install
					Automated Install
					Systemtap Users
				Systemtap Scripts
					Running Systemtap Scripts
					Cross Instrumentation
		System Tuning
			Tuned
				Installation
				Using Tuned
		Summary
Index