Linux Kernel Debugging

Cover
Title Page
Copyright and Credits
Contributors
Table of Contents
Preface
Part 1: A General Introduction and Approaches to Kernel Debugging
Chapter 1: A General Introduction to Debugging Software
	Technical requirements
		Cloning this book's code repository
	Software debugging – what it is, origins, and myths
	Software bugs – a few actual cases
		Patriot missile failure
		The ESA's unmanned Ariane 5 rocket
		Mars Pathfinder reset issue
		The Boeing 737 MAX aircraft – the MCAS and lack of training of the flight crew
		Other cases
	Setting up the workspace
		Running Linux as a native or guest OS
		Running Linux as a guest OS
		Installing the Oracle VirtualBox guest additions
		Installing required software packages
	A tale of two kernels
		A production and a debug kernel
		Setting up our custom production kernel
		Setting up our custom debug kernel
		Seeing the difference – production and debug kernel config
	Debugging – a few quick tips
		A programmer's checklist – seven rules
	Summary
	Further reading
Chapter 2: Approaches to Kernel Debugging
	Technical requirements
	Classifying bug types
		Types of bugs – the classic view
		Types of bugs – the memory view
		Types of bugs – the CVE/CWE security-related view
		Types of bugs – the Linux kernel
	Kernel debugging – why there are different approaches to it
	Summarizing the different approaches to kernel debugging
		The development phase
		Unit testing and/or QA phases
		Categorizing into different scenarios
	Summary
	Further reading
Part 2: Kernel and Driver Debugging Tools and Techniques
Chapter 3: Debug via Instrumentation – printk and Friends
	Technical requirements
	The ubiquitous kernel printk
		Using the printk API's logging levels
		Leveraging the pr_ convenience macros
		Understanding where the printk output goes
		Practically using the printk format specifiers – a few quick tips
	Leveraging the printk for debug purposes
		Writing debug messages to the kernel log
		Debug printing – quick and useful tips
		Device drivers – using the dev_dbg() macro
		Trying our kernel module on the custom production kernel
		Rate limiting the printk
	Using the kernel's powerful dynamic debug feature
		Dynamic debugging via module parameters
		Specifying what and how to print debug messages
		Exercising dynamic debugging on a kernel module on a production kernel
	Remaining printk miscellany
		Printing before console init – the early printk
		Designating the printk to some known presets
		Printing exactly once
		Emitting a printk from userspace
		Easily dumping buffer content
		Remaining points – bootloader log peeking, LED flashing, and more
	Summary
	Further reading
Chapter 4: Debug via Instrumentation – Kprobes
	Understanding kprobes basics
		What we intend to do
	Using static kprobes – traditional approaches to probing
		Demo 1 – static kprobe – trapping into the file open the traditional static kprobes way – simplest case
		Demo 2 – static kprobe – specifying the function to probe via a module parameter
	Understanding the basics of the Application Binary Interface (ABI)
	Using static kprobes – demo 3 and demo 4
		Demo 3 – static kprobe – probing the file open syscall and retrieving the filename parameter
		Demo 4 – semi-automated static kprobe via our helper script
	Getting started with kretprobes
		Kprobes miscellany
	Kprobes – limitations and downsides
		Interface stability
	The easier way – dynamic kprobes or kprobe-based event tracing
		Kprobe-based event tracing – minimal internal details
		Setting up a dynamic kprobe (via kprobe events) on any function
		Using dynamic kprobe event tracing on a kernel module
		Setting up a return probe (kretprobe) with kprobe-perf
	Trapping into the execve() API – via perf and eBPF tooling
		System calls and where they land in the kernel
		Observability with eBPF tools – an introduction
	Summary
	Further reading
Chapter 5: Debugging Kernel Memory Issues – Part 1
	Technical requirements
	What's the problem with memory anyway?
		Tools to catch kernel memory issues – a quick summary
	Using KASAN and UBSAN to find memory bugs
		Understanding KASAN – the basics
		Requirements to use KASAN
		Configuring the kernel for Generic KASAN mode
		Bug hunting with KASAN
		Using the UBSAN kernel checker to find Undefined Behaviour
	Building your kernel and modules with Clang
		Using Clang 13 on Ubuntu 21.10
	Catching memory defects in the kernel – comparisons and notes (Part 1)
		Miscellaneous notes
	Summary
	Further reading
Chapter 6: Debugging Kernel Memory Issues – Part 2
	Technical requirements
	Detecting slab memory corruption via SLUB debug
		Configuring the kernel for SLUB debug
		Leveraging SLUB debug features via the slub_debug kernel parameter
		Running and tabulating the SLUB debug test cases
		Interpreting the kernel's SLUB debug error report
		Learning how to use the slabinfo and related utilities
	Finding memory leakage issues with kmemleak
		Configuring the kernel for kmemleak
		Using kmemleak
		A few tips for developers regarding dynamic kernel memory allocation
	Catching memory defects in the kernel – comparisons and notes (Part 2)
		Miscellaneous notes
	Summary
	Further reading
Chapter 7: Oops! Interpreting the Kernel Bug Diagnostic
	Technical requirements
	Generating a simple kernel bug and Oops
		The procmap utility
		What's this NULL trap page anyway?
		A simple Oops v1 – dereferencing the NULL pointer
		Doing a bit more of an Oops – our buggy module v2
	A kernel Oops and what it signifies
	The devil is in the details – decoding the Oops
		Line-by-line interpretation of an Oops
	Tools and techniques to help determine the location of the Oops
		Using objdump to help pinpoint the Oops code location
		Using GDB to help debug the Oops
		Using addr2line to help pinpoint the Oops code location
		Taking advantage of kernel scripts to help debug kernel issues
		Leveraging the console device to get the kernel log after Oopsing in IRQ context
	An Oops on an ARM Linux system and using netconsole
		Figuring out the actual buggy code location (on ARM)
	A few actual Oopses
	Summary
	Further reading
Chapter 8: Lock Debugging
	Technical requirements
	Locking and lock debugging
	Locking – a quick summarization of key points
		Understanding data races – delving deeper
	Catching concurrency bugs with KCSAN
		What KCSAN does, in a nutshell
		Configuring the kernel for KCSAN
		Using KCSAN
		Knee-jerk reactions to KCSAN reports – please don't!
	A few actual use cases of kernel bugs due to locking defects
		Defects identified by KCSAN
		Identifying locking rules and bugs from the LDV project
		Identifying locking bugs from the Linux kernel Bugzilla
		Identifying some locking defects from various blog articles and the like
	Summary
	Further reading
Part 3: Additional Kernel Debugging Tools and Techniques
Chapter 9: Tracing the Kernel Flow
	Technical requirements
	Kernel tracing technology – an overview
	Using the ftrace kernel tracer
		Accessing ftrace via the filesystem
		Configuring the kernel for ftrace
		Using ftrace to trace the flow of the kernel
		Useful ftrace filtering options
		Case 1 – tracing a single ping with raw ftrace
		Case 2 – tracing a single ping with raw ftrace via the set_event interface
		Using trace_printk() for debugging
		Ftrace – miscellaneous remaining points via FAQs
		Ftrace use cases
	Using the trace-cmd, KernelShark, and perf-tools ftrace frontends
		An introduction to using trace-cmd
		Using the KernelShark GUI
		An introduction to using perf-tools
	An introduction to kernel tracing with LTTng and Trace Compass
		A quick introduction to recording a kernel tracing session with LTTng
		Using the Trace Compass GUI to visualize the single ping LTTng trace
	Summary
	Further reading
Chapter 10: Kernel Panic, Lockups, and Hangs
	Technical requirements
	Panic! – what happens when a kernel panics
		Let's panic
		To the rescue with netconsole
		Interpreting the panic output
		Kernel parameters, tunables, and configs that affect kernel panic
	Writing a custom kernel panic handler routine
		Linux kernel panic notifier chains – the basics
		Setting up our custom panic handler within a module
	Detecting lockups and CPU stalls in the kernel
		A short note on watchdogs
		Employing the kernel's hard and soft lockup detector
	Employing the kernel's hung task and workqueue stall detectors
		Leveraging the kernel hung task detector
		Detecting workqueue stalls
	Summary
	Further reading
Chapter 11: Using Kernel GDB (KGDB)
	Technical requirements
	Conceptually understanding how KGDB works
	Setting up an ARM target system and kernel for KGDB
		Building a minimal custom ARM Linux target system with SEALS
		Configuring the kernel for KGDB
		Testing the target system
	Debugging the kernel with KGDB
		Running our target (emulated) ARM32 system
		Running and working with the remote GDB client on the host system
	Debugging kernel modules with KGDB
		Informing the GDB client about the target module's locations in memory
		Step by step – debugging a buggy module with KGDB
	[K]GDB – a few tips and tricks
		Setting up and using GDB scripts with CONFIG_GDB_SCRIPTS
		KGDB target remote :1234 command doesn't work on physical systems
		Setting the system root with sysroot
		Using GDB's TUI mode
		What to do when the  GDB response occurs
		GDB convenience routines
		GDB custom macros in its startup file
		Fancy breakpoints and hardware watchpoints
		Miscellaneous GDB tips
	Summary
	Further reading
Chapter 12: A Few More Kernel Debugging Approaches
	An introduction to the kdump/crash framework
		Why use kdump/crash?
		Understanding the kdump/crash basic framework
	A mention on performing static analysis on kernel code
		Examples using cppcheck and checkpatch.pl for static analysis
	An introduction to kernel code coverage tools and testing frameworks
		Why is code coverage important?
		A brief note on kernel testing
	Miscellaneous – using journalctl, assertions, and warnings
		Looking up system logs with journalctl
		Assertions, warnings, and BUG() macros
	Summary
	Further reading
Index
Other Books You May Enjoy