Monitoring with Prometheus

Foreword
	Who is this book for?
	Credits and Acknowledgments
	Technical Reviewers
		Jamie Wilkinson
		Paul Gier
	Editor
	Author
	Conventions in the book
	Code and Examples
	Colophon
	Errata
	Disclaimer
	Copyright
	Version
Introduction
	What is monitoring?
		Technology as a customer
		The business as a customer
	Monitoring fundamentals
		Monitoring as afterthought
		Monitoring by rote
		Not monitoring for correctness
		Monitoring statically
		Not monitoring frequently enough
		No automation or self-service
		Good monitoring summary
	Monitoring mechanics
		Probing and introspection
		Pull versus push
		Types of monitoring data
	Metrics
		So what\'s a metric?
		Types of metrics
		Metric summaries
		Metric aggregation
	Monitoring methodologies
		The USE Method
		The Google Four Golden Signals
	Contextual, useful alerts and notifications
	Visualization
	But didn\'t you write that other book?
	What\'s in the book?
	Summary
Introduction to Prometheus
	The Prometheus backstory
	Prometheus architecture
		Metric collection
		Service discovery
		Aggregation and alerting
		Querying data
		Autonomy
		Redundancy and high availability
		Visualization
	The Prometheus data model
		Metric names
		Labels
		Samples
		Notation
		Metrics retention
	Security model
	Prometheus ecosystem
	Useful Prometheus links
	Summary
Installation and Getting Started
	Installing Prometheus
		Installing Prometheus on Linux
		Installing Prometheus on Microsoft Windows
		Alternative Microsoft Windows installation
		Alternative Mac OS X installation
		Stacks
		Installing via configuration management
		Deploying via Kubernetes
	Configuring Prometheus
		Global
		Alerting
		Rule files
		Scrape configuration
	Starting the server
		Running Prometheus via Docker
	First metrics
	Prometheus expression browser
	Time series aggregation
	Capacity planning
		Memory
		Disk
	Summary
Monitoring Nodes and Containers
	Monitoring nodes
		Installing the Node Exporter
		Configuring the Node Exporter
		Configuring the Textfile collector
		Enabling the systemd collector
		Running the Node Exporter
		Scraping the Node Exporter
		Filtering collectors on the server
	Monitoring Docker
		Running cAdvisor
		Scraping cAdvisor
	Scrape lifecycle
	Labels
		Label taxonomies
		Relabelling
	The Node Exporter and cAdvisor metrics
		The trinity and the USE method
		Service status
		Availability and the up metric
		The metadata metric
	Query permanence
		Recording rules
		Configuring recording rules
		Adding recording rules
	Visualization
		Installing Grafana
		Starting and configuring Grafana
		Configuring the Grafana web interface
		First dashboard
	Summary
Service Discovery
	Scrape lifecycle and static configuration redux
	File-based discovery
		Writing files for file discovery
	Inbuilt service discovery plugins
		Amazon EC2 service discovery plugin
	DNS service discovery
	Summary
Alerting and Alertmanager
	Alerting
	How the Alertmanager works
	Installing Alertmanager
		Installing Alertmanager on Linux
		Installing Alertmanager on Microsoft Windows
		Stacks
		Installing via configuration management
	Configuring the Alertmanager
	Running Alertmanager
	Configuring Prometheus for Alertmanager
		Alertmanager service discovery
		Monitoring Alertmanager
	Adding alerting rules
		Adding our first alerting rule
		What happens when an alert fires?
		The alert at the Alertmanager
		Adding new alerts and templates
	Routing
		Routes
	Receivers and notification templates
	Silences and maintenance
		Controlling silences via the Alertmanager
		Controlling silences via amtool
	Summary
Scaling and Reliability
	Reliability and fault tolerance
		Duplicate Prometheus servers
		Setting up Alertmanager clustering
		Configuring Prometheus for an Alertmanager cluster
	Scaling
		Functional scaling
		Horizontal shards
	Remote storage
	Third-party tools
	Summary
Instrumenting Applications
	An application monitoring primer
		Where should I instrument?
		Instrument taxonomies
	Metrics
		Application metrics
		Business metrics
		Where to put your metrics
		The utility pattern
		The external pattern
		Building metrics into a sample application
	Summary
Logging as Instrumentation
	Processing logs for metrics
	Introducing mtail
		Installing mtail
		Using mtail
		Running mtail
	Processing web server access logs
	Parsing Rails logs into a histogram
	Deploying mtail
	Scraping our mtail endpoint
	Summary
Probing
	Probing architecture
	The blackbox exporter
	Installing the exporter
		Installing the exporter on Linux
		Installing the exporter on Microsoft Windows
		Installing via configuration management
	Configuring the exporter
		HTTP check
		ICMP check
		DNS check
	Starting the exporter
	Creating the Prometheus job
	Summary
Pushing Metrics and the Pushgateway
	The Pushgateway
		When not to use the Pushgateway
		Installing the Pushgateway
		Installing the Pushgateway on Linux
		Installing the Pushgateway on Microsoft Windows
		Installing via configuration management
		Configuring and running the Pushgateway
		Sending metrics to the Pushgateway
		Viewing metrics on the Pushgateway
		Deleting metrics in the Pushgateway
		Sending metrics from a client
	Summary
Monitoring a Stack - Kubernetes
	Our Kubernetes cluster
	Running Prometheus on Kubernetes
	Monitoring Kubernetes
	Monitoring our Kubernetes nodes
		Node Exporter DaemonSet
		Node Exporter service
		Deploying the Node Exporter
		The Node Exporter job
		Node Explorer rules
	Kubernetes
		Kube-state-metrics
		Kube API
		CAdvisor and Nodes
	Summary
Monitoring a Stack - Tornado
	Sidecar pattern
	MySQL
		MySQL Prometheus configuration
	Redis
		Redis Prometheus configuration
	Tornado
		Adding the Clojure wrapper
		Adding a registry
		Adding metrics
		Exporting the metrics
		Tornado Prometheus configuration
	Summary
List of Figures
List of Listings
Index