Logs and Telemetry: Using Fluent Bit, Kubernetes, streaming and more

Logs and Telemetry
brief contents
contents
foreword
preface
acknowledgments
about this book
	How this book is organized: A road map
	About the code
	liveBook discussion forum
about the author
about the cover illustration
Part 1—From concepts to running Fluent Bit
	1 Introduction to Fluent Bit
		1.1 Why is Fluent Bit so important?
			1.1.1 The value of event distribution
			1.1.2 Fluent’s place in CNCF
		1.2 Core Fluent Bit concepts
			1.2.1 Payload structure
			1.2.2 Logical architecture
		1.3 Drivers of Fluent Bit adoption
			1.3.1 Small footprint, efficiency, and speed
			1.3.2 Effect of OpenTelemetry and how Fluent Bit relates to It
			1.3.3 Extending Fluent Bit with C, Go, WebAssembly, and Lua
			1.3.4 Fluent Bit and stream processing
			1.3.5 OTel vs. Fluent Bit and Fluentd
		1.4 Is Fluent Bit a child or a successor of Fluentd?
		1.5 How we’re going to discover Fluent Bit
			1.5.1 How much Kubernetes will this book involve?
			1.5.2 Logging in Action
		Summary
	2 From zero to “Hello, World”
		2.1 Multiple ways to configure Fluent Bit
			2.1.1 Configuration formats
			2.1.2 CLI controls
			2.1.3 Defining a monitoring pipeline using the CLI
			2.1.4 Fluent Bit prebuilt Docker container
		2.2 Fluent Bit configuration in two forms
			2.2.1 Fluent Bit vs. Fluentd configuration comparison
			2.2.2 Comparing Classic and YAML configuration
		2.3 Checking configuration with a dry run
			2.3.1 Exercise: Using - -dry-run to help fix a conf file
		2.4 Configuring file inclusions
			2.4.1 Creating dynamic configuration by using inclusions
			2.4.2 Proving stub inclusions
		2.5 Environment variables in the configuration
			2.5.1 Applying environment variables
			2.5.2 Setting environment variables
		2.6 Monitoring Fluent Bit’s health
		Summary
Part 2—Digging deeper
	3 Capturing inputs
		3.1 Fluent Bit plugins
		3.2 OS and device sources
			3.2.1 Monitoring infrastructure with native executables
			3.2.2 Tuning monitoring sources
			3.2.3 Device sources
		3.3 Using stdout
			3.3.1 The twelve-factor app and Fluent Bit
			3.3.2 Running the containerized Log Simulator
		3.4 File-based log events
		3.5 Capturing log files
			3.5.1 Simple file consumption
			3.5.2 Supporting long-running processes
			3.5.3 Capturing logs from short-lived applications
		3.6 Network events and communication between Fluent Bit and Fluentd
			3.6.1 Network input sources
			3.6.2 HTTP source
			3.6.3 Securing communication with SSL/TLS
			3.6.4 forward source
			3.6.5 Beyond network ports
			3.6.6 Internode communication
			3.6.7 OpenTelemetry
		3.7 Fluent Bit buffers and chunks
		3.8 Other sources
			3.8.1 Container-related plugins
			3.8.2 Getting data from other processes
			3.8.3 Observing the observers
		Summary
	4 Getting inputs from containers and Kubernetes
		4.1 Architectural context
		4.2 Fluent Bit capturing Docker events and metrics
			4.2.1 Docker Events
			4.2.2 Docker Metrics
		4.3 Using Podman as a Docker alternative
		4.4 Other containers
		4.5 Container logging drivers
		4.6 Application direct to Fluent Bit
			4.6.1 OpenTelemetry’s approach to containerized applications
			4.6.2 Deploying for application direct logging
			4.6.3 Enriching log events with Pod context by injection
			4.6.4 Enriching log events with Pod context by filter
		4.7 Kubernetes and observability
			4.7.1 Understanding Kubernetes’ position on logging
			4.7.2 Kubernetes auditing
			4.7.3 Kubernetes events input
			4.7.4 The many parts of the Kubernetes ecosystem
			4.7.5 Container Images
			4.7.6 Helm charts
		4.8 Kubernetes operator
		4.9 Observations on Fluent Bit with Kubernetes
		4.10 The next frontier of observability with Fluent Bit: eBPF
		Summary
	5 Outputting events
		5.1 Architectural context
		5.2 Common characteristics of Fluent Bit output plugins
			5.2.1 Output resilience through retries
			5.2.2 Network controls
			5.2.3 Worker threads
			5.2.4 Considerations for using threads
		5.3 Null output
			5.3.1 Monitoring with Fluent Bit
			5.3.2 Configuring null output
		5.4 Sending log events to the console
			5.4.1 Formatting outputs
			5.4.2 Seeing matching at work
		5.5 Writing to files
		5.6 Prometheus outputs
			5.6.1 Prometheus Node Exporter
			5.6.2 Running our Prometheus configuration
			5.6.3 Prometheus Fluent Bit Exporter
			5.6.4 Prometheus remote writer
		5.7 PostgreSQL output
		5.8 HTTP output
		5.9 Forwarding to other Fluent nodes
		5.10 OpenTelemetry
		5.11 Hyperscaler native and SaaS observability
		Summary
	6 Parsing to extract more meaning
		6.1 Architectural context
		6.2 The goal of parsing
		6.3 Relationship between parsers and filters
		6.4 Prebuilt parsers
		6.5 Parsing an Apache log file
		6.6 Multiline parsing
		6.7 Custom parsing
		6.8 Processing JSON
			6.8.1 Changing the log event timestamp
			6.8.2 Diagnosing the unhappy paths
		6.9 Other types of parsers
			6.9.1 logfmt
			6.9.2 LTSV
		6.10 Decoders
		6.11 Parsing shortcut for file inputs
		Summary
	7 Filtering and transforming events
		7.1 Architectural context
		7.2 Integrating and enriching with filters
			7.2.1 Directing and securing logs with GeoIP
			7.2.2 Using the CheckList filter
		7.3 Extending and amending with filters
			7.3.1 Taking a brief look at the nest filter
			7.3.2 Illustrating the record_modifier filter
			7.3.3 Illustrating the modify filter
			7.3.4 Bringing it together
			7.3.5 Testing filters
		7.4 Routing and controlling
			7.4.1 Using the record accessor
			7.4.2 Rewriting the tag filter example
			7.4.3 Explicitly including and excluding events with grep
		7.5 Controlling events
			7.5.1 throttle
			7.5.2 log_to_metrics
			7.5.3 Advanced use of matching
		7.6 Custom filtering with Lua
			7.6.1 Background of Lua
			7.6.2 Implementing a Lua filter
		Summary
Part 3—Plugins and queries
	8 Stream processors for time series calculations and filtering
		8.1 Architectural context
		8.2 Key ideas
		8.3 Basic query
		8.4 Stream-processing windows
			8.4.1 Hopping windows
			8.4.2 Tumbling windows
			8.4.3 Setting window durations
			8.4.4 Deciding which window to use
		8.5 Selecting multiple attributes and naming
		8.6 Streams vs. tags
		8.7 Creating streams
		8.8 Chaining and creating new streams
		8.9 Typical use cases for streaming
			8.9.1 Forecasting
			8.9.2 Intermittent error tolerance
			8.9.3 Spurious data values
			8.9.4 Absence of events
			8.9.5 Cross-referencing streams
		Summary
	9 Building processors and Fluent Bit extension options
		9.1 Architectural context
		9.2 Fluent Bit processor: Changing the behavior of existing plugins
			9.2.1 Processor with Lua for logs
			9.2.2 Content modifier processor
			9.2.3 Processor for traces
			9.2.4 Processor to metrics
			9.2.5 Processor using SQL
		9.3 Why we need to extend Fluent Bit
		9.4 C language
			9.4.1 Considerations
			9.4.2 Benefits
			9.4.3 Drawbacks
			9.4.4 Tools for the job
		9.5 Go language
			9.5.1 Benefits
			9.5.2 Drawbacks
		9.6 WebAssembly
			9.6.1 Benefits
			9.6.2 Drawbacks
		9.7 Selecting an extension strategy
		Summary
	10 Building plugins
		10.1 Architectural context
		10.2 Why Go?
		10.3 Plugin objective
		10.4 Go plugin approach
			10.4.1 Simplifying our build process
			10.4.2 Code structure
			10.4.3 Fluent Bit feature switches
			10.4.4 The build process for plugins
		10.5 Understanding the plugin life cycle
			10.5.1 Input life cycle
			10.5.2 Output life cycle
		10.6 Implementing the plugin
			10.6.1 Setting up MySQL
			10.6.2 Input plugin
			10.6.3 Building the code
			10.6.4 Output plugin
		10.7 Deploying the custom plugin
		10.8 Configuring our scenario
		10.9 Executing the build
		10.10 Running the custom plugins
		Summary
	11 Putting Fluent Bit into action: An enterprise use case
		11.1 Use case
		11.2 Deployment needs
		11.3 Customer dashboards
			11.3.1 Customer dashboards with Fluent Bit
			11.3.2 Customer dashboard containers
			11.3.3 Customer dashboard innovation
		11.4 Development pipelines
		11.5 Core services
		11.6 Central accounting needs
		11.7 Operational processes
		11.8 Tool choices
		11.9 Conclusion
		Summary
appendix A—Installations
	A.1 Tool installation overview
	A.2 Downloading book resources
	A.3 Prepping Linux
	A.4 Fluent Bit
		A.4.1 Linux Installs
		A.4.2 macOS
		A.4.3 Windows installs
	A.5 Docker
		A.5.1 Windows
		A.5.2 Verifying the installation
		A.5.3 Linux (including macOS)
		A.5.4 macOS
	A.6 Kubernetes
	A.7 LogSimulator
		A.7.1 Running as a downloaded image
		A.7.2 Running as a locally built Docker image
		A.7.3 Java and Groovy
		A.7.4 Post-LogSimulator use
	A.8 WireMock
	A.9 Postman
	A.10 Postgres
	A.11 MySQL
	A.12 Prometheus
	A.13 jq
appendix B—Useful resources
	B.1 Standard plugins based on platform
		B.1.1 Input plugins
		B.1.2 Output plugins
		B.1.3 Filter plugins
		B.1.4 Processors
	B.2 Predefined parsers
		B.2.1 parser.conf file
		B.2.2 parsers_ambassador file
		B.2.3 parsers_cinder file
		B.2.4 parsers_extra
		B.2.5 parsers_java file
		B.2.6 parsers_kafka file
		B.2.7 parsers_openstack file
	B.3 Multiline parsers
	B.4 Sources of predefined regular expressions
	B.5 Plugins supporting record accessor
	B.6 Stream processor functions
	B.7 Reserved attribute names
	B.8 Expressing time
	B.9 Expressing data sizes
	B.10 Fluent Bit formatters
	B.11 Useful third-party tools
	B.12 Observability
	B.13 Helpful logging practices and resources
	B.14 Additional reading
	B.15 Web resources
		B.15.1 Formal and de facto standards
		B.15.2 Additional web resources
	B.16 Fluent Bit resources
	B.17 Lua
	B.18 WASM and WASI
	B.19 C development resources
	B.20 Logging format definitions
appendix C—Comparing Fluent Bit and Fluentd
	C.1 Technology differences
	C.2 Configuration capabilities
	C.3 Inputs and outputs
		C.3.1 Support for logging frameworks
		C.3.2 Plugin choice
		C.3.3 Secondary/fallback output options
		C.3.4 OpenTelemetry
		C.3.5 Customization with embedded code
	C.4 Routing
	C.5 Buffering and internal data structure
	C.6 Streaming processing
	C.7 Conclusion
index
	A
	B
	C
	D
	E
	F
	G
	H
	I
	J
	K
	L
	M
	N
	O
	P
	Q
	R
	S
	T
	U
	V
	W
	Y