Learning Elastic Stack 7.0 : distributed search, analytics, and visualization using Elasticsearch, Logstash, Beats, and Kibana

Cover
Title Page
Copyright and Credits
About Packt
Contributors
Table of Contents
Preface
Section 1: Introduction to Elastic Stack and Elasticsearch
Chapter 1: Introducing Elastic Stack
	What is Elasticsearch, and why use it?
		Schemaless and document-oriented
		Searching capability
		Analytics
		Rich client library support and the REST API
		Easy to operate and easy to scale 
		Near real-time capable
		Lightning–fast
		Fault-tolerant
	Exploring the components of the Elastic Stack
		Elasticsearch
		Logstash
		Beats
		Kibana
		X-Pack
			Security
			Monitoring
			Reporting
			Alerting
			Graph
			Machine learning
		Elastic Cloud
	Use cases of Elastic Stack
		Log and security analytics
		Product search
		Metrics analytics
		Web search and website search
	Downloading and installing
		Installing Elasticsearch
		Installing Kibana
	Summary
Chapter 2: Getting Started with Elasticsearch
	Using the Kibana Console UI
	Core concepts of Elasticsearch
		Indexes
		Types
		Documents
		Nodes
		Clusters
		Shards and replicas
		Mappings and datatypes
			Datatypes
				Core datatypes
				Complex datatypes
				Other datatypes
			Mappings
				Creating an index with the name catalog
				Defining the mappings for the type of product
		Inverted indexes
	CRUD operations
		Index API
			Indexing a document by providing an ID
			Indexing a document without providing an ID
		Get API
		Update API
		Delete API
	Creating indexes and taking control of mapping
		Creating an index
		Creating type mapping in an existing index
		Updating a mapping
	REST API overview
		Common API conventions
			Formatting the JSON response
			Dealing with multiple indexes
				Searching all documents in one index
				Searching all documents in multiple indexes
				Searching all the documents of a particular type in all indexes
	Summary
Section 2: Analytics and Visualizing Data
Chapter 3: Searching - What is Relevant
	The basics of text analysis
		Understanding Elasticsearch analyzers
			Character filters
			Tokenizer
				Standard tokenizer
			Token filters
		Using built-in analyzers
			Standard analyzer
		Implementing autocomplete with a custom analyzer
	Searching from structured data
		Range query
			Range query on numeric types
			Range query with score boosting
			Range query on dates
		Exists query
		Term query
	Searching from the full text
		Match query
			Operator
			Minimum should match
			Fuzziness
		Match phrase query
		Multi match query
			Querying multiple fields with defaults
			Boosting one or more fields
			With types of multi match queries
	Writing compound queries
		Constant score query
		Bool query
			Combining OR conditions
			Combining AND and OR conditions
			Adding NOT conditions
	Modeling relationships
		has_child query
		has_parent query
		parent_id query
	Summary
Chapter 4: Analytics with Elasticsearch
	The basics of aggregations
		Bucket aggregations
		Metric aggregations
		Matrix aggregations
		Pipeline aggregations
	Preparing data for analysis
		Understanding the structure of the data
		Loading the data using Logstash
	Metric aggregations
		Sum, average, min, and max aggregations
			Sum aggregation
			Average aggregation
			Min aggregation
			Max aggregation
		Stats and extended stats aggregations
			Stats aggregation
			Extended stats aggregation
		Cardinality aggregation
	Bucket aggregations
		Bucketing on string data
			Terms aggregation
		Bucketing on numerical data
			Histogram aggregation
			Range aggregation
		Aggregations on filtered data
		Nesting aggregations
		Bucketing on custom conditions
			Filter aggregation
			Filters aggregation
		Bucketing on date/time data
			Date Histogram aggregation
				Creating buckets across time periods
				Using a different time zone
				Computing other metrics within sliced time intervals
				Focusing on a specific day and changing intervals
		Bucketing on geospatial data
			Geodistance aggregation
			GeoHash grid aggregation
	Pipeline aggregations
		Calculating the cumulative sum of usage over time
	Summary
Chapter 5: Analyzing Log Data
	Log analysis challenges
	Using Logstash
		Installation and configuration
			Prerequisites
			Downloading and installing Logstash
				Installing on Windows
				Installing on Linux
		Running Logstash
	The Logstash architecture
	Overview of Logstash plugins
		Installing or updating plugins
			Input plugins
			Output plugins
			Filter plugins
			Codec plugins
		Exploring plugins
			Exploring input plugins
				File
				Beats
				JDBC
				IMAP
			Output plugins
				Elasticsearch
				CSV
				Kafka
				PagerDuty
			Codec plugins
				JSON
				Rubydebug 
				Multiline
			Filter plugins
	Ingest node
		Defining a pipeline 
		Ingest APIs
			Put pipeline API
			Get pipeline API
			Delete pipeline API
			Simulate pipeline API
	Summary
Chapter 6: Building Data Pipelines with Logstash
	Parsing and enriching logs using Logstash
		Filter plugins
			CSV filter 
			Mutate filter
			Grok filter
			Date filter
			Geoip filter
			Useragent filter
	Introducing Beats
		Beats by Elastic.co
			Filebeat
			Metricbeat
			Packetbeat
			Heartbeat
			Winlogbeat
			Auditbeat
			Journalbeat
			Functionbeat
		Community Beats
		Logstash versus Beats
	Filebeat
		Downloading and installing Filebeat
			Installing on Windows
			Installing on Linux
		Architecture
		Configuring Filebeat
			Filebeat inputs
			Filebeat general/global options
			Output configuration 
			Logging
			Filebeat modules
	Summary
Chapter 7: Visualizing Data with Kibana
	Downloading and installing Kibana
		Installing on Windows
		Installing on Linux
		Configuring Kibana
	Preparing data
	Kibana UI
		User interaction
		Configuring the index pattern
		Discover
			Elasticsearch query string/Lucene query
			Elasticsearch DSL query
			KQL
		Visualize
			Kibana aggregations
				Bucket aggregations
				Metric
		Creating a visualization
		Visualization types
			Line, area, and bar charts
			Data tables
			Markdown widgets
			Metrics
			Goals
			Gauges
			Pie charts
			Co-ordinate maps
			Region maps
			Tag clouds
		Visualizations in action
			Response codes over time
			Top 10 requested URLs
			Bandwidth usage of the top five countries over time
			Web traffic originating from different countries
			Most used user agent
		Dashboards
			Creating a dashboard
			Saving the dashboard 
			Cloning the dashboard
			Sharing the dashboard 
	Timelion
		Timelion 
		Timelion expressions
	Using plugins
		Installing plugins
		Removing plugins
	Summary
Section 3: Elastic Stack Extensions
Chapter 8: Elastic X-Pack
	[Installation]
		Installation
		Activating X-Pack trial account
			Generating passwords for default users
	Configuring X-Pack
	Securing Elasticsearch and Kibana
		User authentication
		User authorization
		Security in action
			Creating a new user
				Deleting a user
				Changing the password
			Creating a new role
				Deleting or editing a role
			Document-level security or field-level security
			X-Pack security APIs
				User Management APIs
				Role Management APIs
	Monitoring Elasticsearch
		Monitoring UI
			Elasticsearch metrics
				Overview tab
				Nodes tab
				The Indices tab
	Alerting
		Anatomy of a watch
		Alerting in action
			Creating a new alert
				Threshold Alert
				Advanced Watch
			Deleting/deactivating/editing a watch
	Summary
Section 4: Production and Server Infrastructure
Chapter 9: Running Elastic Stack in Production
	Hosting Elastic Stack on a managed cloud
		Getting up and running on Elastic Cloud
		Using Kibana
		Overriding configuration 
		Recovering from a snapshot
	Hosting Elastic Stack on your own
		Selecting hardware
		Selecting an operating system
		Configuring Elasticsearch nodes
			JVM heap size
			Disable swapping
			File descriptors
			Thread pools and garbage collector
		Managing and monitoring Elasticsearch
		Running in Docker containers
		Special considerations while deploying to a cloud
			Choosing instance type
			Changing default ports; do not expose ports!
			Proxy requests
			Binding HTTP to local addresses
			Installing EC2 discovery plugin
			Installing the S3 repository plugin
			Setting up periodic snapshots
	Backing up and restoring
		Setting up a repository for snapshots
			Shared filesystem
		Cloud or distributed filesystems
		Taking snapshots
		Restoring a specific snapshot
	Setting up index aliases
		Understanding index aliases
		How index aliases can help
	Setting up index templates
		Defining an index template
		Creating indexes on the fly
	Modeling time series data
		Scaling the index with unpredictable volume over time
			Unit of parallelism in Elasticsearch
				The effect of the number of shards on the relevance score
				The effect of the number of shards on the accuracy of aggregations
		Changing the mapping over time
			New fields get added
			Existing fields get removed
		Automatically deleting older documents
		How index-per-timeframe solves these issues
			Scaling with index-per-timeframe
			Changing the mapping over time
			Automatically deleting older documents
	Summary
Chapter 10: Building a Sensor Data Analytics Application
	Introduction to the application
		Understanding the sensor-generated data
		Understanding the sensor metadata
		Understanding the final stored data
	Modeling data in Elasticsearch
		Defining an index template
		Understanding the mapping
	Setting up the metadata database
	Building the Logstash data pipeline
		Accepting JSON requests over the web
		Enriching the JSON with the metadata we have in the MySQL database
			The jdbc_streaming plugin 
			The mutate plugin
				Moving the looked-up fields that are under lookupResult directly in JSON
				Combining the latitude and longitude fields under lookupResult as a location field
				Removing the unnecessary fields
		Store the resulting documents in Elasticsearch
	Sending data to Logstash over HTTP
	Visualizing the data in Kibana
		Setting up an index pattern in Kibana
		Building visualizations
			How does the average temperature change over time?
			How does the average humidity change over time?
			How do temperature and humidity change at each location over time?
			Can I visualize temperature and humidity over a map?
			How are the sensors distributed across departments?
		Creating a dashboard
	Summary
Chapter 11: Monitoring Server Infrastructure
	Metricbeat
		Downloading and installing Metricbeat
			Installing on Windows
			Installing on Linux
		Architecture
			Event structure
	Configuring Metricbeat
		Module configuration
			Enabling module configs in the modules.d directory
			Enabling module configs in the metricbeat.yml file
		General settings
		Output configuration 
		Logging
	Capturing system metrics
		Running Metricbeat with the system module
		Specifying aliases
		Visualizing system metrics using Kibana
	Deployment architecture
	Summary
Other Books You May Enjoy
Index