Applied ELK Stack: Data Insights and Business Metrics With Collective Capability of ElasticSearch, Logstash and Kibana

CHAPTER 1: Introduction to ELK Stack
Log Analysis in Today's World
The ELK Stack
Elasticsearch
Logstash
Kibana
ELK Data Pipeline
ELK Stack Installation
Installing Elasticsearch
Running Elasticsearch
Elasticsearch Configuration and Settings
Environment Variables
System Configuration
Installing Elasticsearch Plugins
Installing Logstash
Running Logstash
Logstash with Elasticsearch Output
Configuring Logstash
Installing Logstash Forwarder
Extending Logstash Functionality
Installing Kibana
Kibana Configuration
Kibana Interface
Summary
Sample Dataset
Data Format
Logstash Configuration
Comments
Configuring for Events
Field References
Conditionals
Metadata
Filtering Events
Shipping Events
Reloading Configuration File
Multiline Event Configuration
Analyzing Events
Data Visualization
Building a Line Chart
Building a Bar Chart
Building a Metric
Building a Data Table
Summary
CHAPTER 3: Extending Logstash
Plugin Management
Download and Installation
Plugin Installation
Updating Plugin
Uninstallation
Plugin Structure
Prerequisite
Basic Structure
Configuration
Setup
Execution
Teardown
Build Custom Plugin
Plugin Packaging
Summary
Ubiquity of Data
Elasticsearch Cluster
Node
Anatomy of a Document
Metadata Information
Index
Type
Id
Shard
Primary and Replica Shard
Elasticsearch API
Cluster Health and Configuration
Index Management
Specify Id
Custom Id
Auto-Generated Id
Document Management
Document Retrieval
Partial Document Retrieval
Document Existence
Multiple Document Retrieval
Document Updates
Updating Documents Partially
Partial Updates with Scripts
Conflicting Updates
Document Creation
Document Deletion
Bulk Operations
Bulk Request Size
Conflict Management
Summary
Search Your Way
Simple Searches
Search Without Parameter
hits
took
shards
timeout
Multi-Index, Multi-Type
Pagination
Search Lite
The _all Field
Query Mashup
Query DSL
Query Clause Construction
Working with Multiple Clauses
Filter and Query
Performance Concerns
Key Filters and Queries
term Filter
terms Filter
range Filter
exists and missing Filters
bool Filter
match_all Query
match Query
multi_match Query
bool Query
Filter-Query Combination
Filtering a Query
Only Filter
Filter via Query
Query Validation
Error Diagnostics
Summary
CHAPTER 6: Mapping and Analysis
Data Mapping and Analysis
Exact Values and Full Text
Inverted Index
Data Analysis
Prepackaged Analyzers
When to Use Analyzers
You can Test Analyzers
Assign Analyzer
Data Mapping
Simple Field Types
Observe the Mapping
Mapping Customization
index
analyzer
Mapping Revision
Mapping Test
Complex Field Types
Multi-value Fields
Empty Fields
Multi-level Objects
Indexing Inner Objects
Inner Object Arrays
Summary
Aggregation Basics
Buckets
Metrics
The Two Together
Fun with Aggregation
Metrics to the Rescue
Buckets within Buckets
Multiple Metrics
Data Visualization with Bar Charts
Time Series Aggregations
Multi-Tier Correlation
Aggregation Scoping
Global Bucket
Aggregations with Query Filters
Query with Filter
Filter Bucket
Post Filter
Multivalue Bucket Sorting
Intrinsic Sorts
Metric Based Sorting
Summary
Introducing Kibana
Kibana Features
Kibana User Interface
Discover Page
Time Filter
Query and Search Data
Free Text Search
Field Searches
Range Searches
New Search
Saving Search
Field Search Using Field List
Summary
CHAPTER 9: Kibana - Data Visualization
Visualize Page
Metrics and Buckets Aggregations
Buckets
Metrics
Advanced Options
Choosing Search Data Source
Visualization Canvas
Toolbar
Aggregation Designer
Preview Canvas
Building Visualization
Visualization Types
Area Chart
Data Table
Line Chart
Markdown Widget
Metric
Pie Chart
Tile Map
Heat Map
Desaturate Map Tiles
Vertical Bar Chart
Summary
Introduction to Dashboard Page
Working with the Toolbar
New Dashboard Option
Adding Visualizations
Search Bar
Save Dashboard
Load Saved Dashboard
Sharing the Saved Dashboard
Working with Dashboard Canvas
Move Visualization
Resize Visualization
Edit Visualization
Remove Visualization
Embed Dashboard in Web Page
Debug Panel
Table
Request
Response
Statistics
Summary
CHAPTER 11: Designing for Scale
Elasticsearch Cluster for Scale
Adding Nodes to Cluster
Discovering Cluster Nodes
Multicast Discovery
Unicast Discovery
Master Node Election
Fault Detection
Removal of Nodes from Cluster
Decommissioning of Nodes
Upgrading Elasticsearch Nodes
Rolling Restart
Quick Restart
Cluster Information
Scaling Options
Over-Sharding
Data Slicing
Maximizing Throughput
Aliases
Working with Alias
Benefits of Aliases
Living with Aliases
Creating Aliases
Camouflaging Documents with Filters
Routing
Significance of Routing
Routing Strategies
Determining Shards
Routing Configuration
Routing in Combination with Aliases
Summary
CHAPTER 12: ELK Stack in Production
Deployment Considerations
Memory
Disks
Network
Java Virtual Machine
Data Management
Request Grouping
Bulk Indexing, Updating and Deleting
Multisearch and Multiget APIs
Elasticsearch Tuning
Lucene Segment Optimization
Thresholds for Refresh and Flush
Merge Policies
Store Throttling
Cache Management
Filter Caches
Shard Query Cache
JVM Heap and OS Cache
Warmers for Caches
Configuration Management
Better than Defaults
Index Templates
Monitoring and Troubleshooting
Health of the Cluster
Detect Index Problem
Examining Individual Nodes
indices Section
OS and Process Sections
JVM Section
Threadpool Section
F5 and Network Sections
Circuit Breaker
Cluster Statistics
Index Statistics
Pending Tasks
Logging
Slowlog
Rolling Restarts
Backup and Restore
Cluster Backup
Creating the Repository
Snapshot of All Open Indices
Snapshot of Particular Index
Listing Snapshot of Information
Snapshot Deletion
Monitoring Snapshot Progress
Cancelling Snapshot
Restore from Snapshot
Monitor Restore Operations
Cancelling Restore
Summary