Hands-On Software Engineering with Golang: Move beyond basic programming to design and build reliable software with clean code

Title Page
Copyright and Credits
Dedication
About Packt
Contributors
Table of Contents
Preface
Section 1: Software Engineering and the Software Development Life Cycle
Chapter 1: A Bird's-Eye View of Software Engineering
	What is software engineering?
	Types of software engineering roles
		The role of the software engineer (SWE)
		The role of the software development engineer in test (SDET)
		The role of the site reliability engineer (SRE)
		The role of the release engineer (RE)
		The role of the system architect
	A list of software development models that all engineers should know
		Waterfall
		Iterative enhancement
		Spiral
		Agile
			Lean
				Eliminate waste
				Create knowledge
				Defer commitment
				Build in quality
				Deliver fast
				Respect and empower people
				See and optimize the whole
			Scrum
				Scrum roles
				Essential Scrum events
			Kanban
		DevOps
			The CAMS model
			The three ways model
	Summary
	Questions
	Further reading
Section 2: Best Practices for Maintainable and Testable Go Code
Chapter 2: Best Practices for Writing Clean and Maintainable Go Code
	The SOLID principles of object-oriented design
		Single responsibility
		Open/closed principle
		Liskov substitution
		Interface segregation
		Dependency inversion
		Applying the SOLID principles
	Organizing code into packages
		Naming conventions for Go packages
		Circular dependencies
			Breaking circular dependencies via implicit interfaces
			Sometimes, code repetition is not a bad idea!
	Tips and tools for writing lean and easy-to-maintain Go code
		Optimizing function implementations for readability
		Variable naming conventions
		Using Go interfaces effectively
		Zero values are your friends
		Using tools to analyze and manipulate Go programs
			Taking care of formatting and imports (gofmt, goimports)
			Refactoring code across packages (gorename, gomvpkg, fix)
			Improving code quality metrics with the help of linters
	Summary
	Questions
	Further reading
Chapter 3: Dependency Management
	What's all the fuss about software versioning?
		Semantic versioning
			Comparing semantic versions
			Applying semantic versioning to Go packages
		Managing the source code for multiple package versions
			Single repository with versioned folders
			Single repository – multiple branches
	Vendoring – the good, the bad, and the ugly
		Benefits of vendoring dependencies
		Is vendoring always a good idea?
	Strategies and tools for vendoring dependencies
		The dep tool
			The Gopkg.toml file
			The Gopkg.lock file
		Go modules – the way forward
		Fork packages
	Summary
	Questions
	Further reading
Chapter 4: The Art of Testing
	Technical requirements
	Unit testing
		Mocks, stubs, fakes, and spies – commonalities and differences
			Stubs and spies!
			Mocks
				Introducing gomock
				Exploring the details of the project we want to write tests for
				Leveraging gomock to write a unit test for our application
			Fake objects
		Black-box versus white-box testing for Go packages – an example
			The services behind the facade
			Writing black-box tests
			Boosting code coverage via white-box tests
		Table-driven tests versus subtests
			Table-driven tests
			Subtests
			The best of both worlds
		Using third-party testing frameworks
	Integration versus functional testing
		Integration tests
		Functional tests
		Functional tests part deux – testing in production!
	Smoke tests
	Chaos testing – breaking your systems in fun and interesting ways!
	Tips and tricks for writing tests
		Using environment variables to set up or skip tests
		Speeding up testing for local development
		Excluding classes of tests via build flags
		This is not the output you are looking for – mocking calls to external binaries
		Testing timeouts is easy when you have all the time in the world!
	Summary
	Questions
	Further reading
Section 3: Designing and Building a Multi-Tier System from Scratch
Chapter 5: The Links 'R'; Us Project
	System overview – what are we going to be building?
	Selecting an SDLC model for our project
		Iterating faster using an Agile framework
		Elephant carpaccio – how to iterate even faster!
	Requirements analysis
		Functional requirements
			User story – link submission
			User story – search
			User story – crawl link graph
			User story – calculate PageRank scores
			User story – monitor Links 'R' Us health
		Non-functional requirements
			Service-level objectives
			Security considerations
			Being good netizens
	System component modeling
		The crawler
			The link filter
			The link fetcher
			The content extractor
			The link extractor
		The content indexer
		The link provider
		The link graph
		The PageRank calculator
		The metrics store
		The frontend
	Monolith or microservices? The ultimate question
	Summary
	Questions
	Further reading
Chapter 6: Building a Persistence Layer
	Technical requirements
		Running tests that require CockroachDB
		Running tests that require Elasticsearch
	Exploring a taxonomy of database systems
		Key-value stores
		Relational databases
		NoSQL databases
		Document databases
	Understanding the need for a data layer abstraction
	Designing the data layer for the link graph component
		Creating an ER diagram for the link graph store
		Listing the required set of operations for the data access layer
		Defining a Go interface for the link graph
			Partitioning links and edges for processing the graph in parallel
			Iterating Links and Edges
		Verifying graph implementations using a shared test suite
		Implementing an in-memory graph store
			Upserting links
			Upserting edges
			Looking up links
			Iterating links/edges
			Removing stale edges
			Setting up a test suite for the graph implementation
		Scaling across with a CockroachDB-backed graph implementation
			Dealing with DB migrations
			An overview of the DB schema for the CockroachDB implementation
			Upserting links
			Upserting edges
			Looking up links
			Iterating links/edges
			Removing stale edges
			Setting up a test suite for the CockroachDB implementation
	Designing the data layer for the text indexer component
		A model for indexed documents
		Listing the set of operations that the text indexer needs to support
			Defining the Indexer interface
		Verifying indexer implementations using a shared test suite
		An in-memory Indexer implementation using bleve
			Indexing documents
			Looking up documents and updating their PageRank score
			Searching the index
			Iterating the list of search results
			Setting up a test suite for the in-memory indexer
		Scaling across an Elasticsearch indexer implementation
			Creating a new Elasticsearch indexer instance
			Indexing and looking up documents
			Performing paginated searches
			Updating the PageRank score for a document
			Setting up a test suite for the Elasticsearch indexer
	Summary
	Questions
	Further reading
Chapter 7: Data-Processing Pipelines
	Technical requirements
	Building a generic data-processing pipeline in Go
		Design goals for the pipeline package
		Modeling pipeline payloads
		Multistage processing
			Stageless pipelines – is that even possible?
		Strategies for handling errors
			Accumulating and returning all errors
			Using a dead-letter queue
			Terminating the pipeline's execution if an error occurs
		Synchronous versus asynchronous pipelines
			Synchronous pipelines
			Asynchronous pipelines
		Implementing a stage worker for executing payload processors
			FIFO
			Fixed and dynamic worker pools
			1-to-N broadcasting
		Implementing the input source worker
		Implementing the output sink worker
		Putting it all together – the pipeline API
	Building a crawler pipeline for the Links 'R' Us project
		Defining the payload for the crawler
		Implementing a source and a sink for the crawler
		Fetching the contents of graph links 
		Extracting outgoing links from retrieved webpages
		Extracting the title and text from retrieved web pages
		Inserting discovered outgoing links to the graph
		Indexing the contents of retrieved web pages
		Assembling and running the pipeline
	Summary
	Questions
	Further reading
Chapter 8: Graph-Based Data Processing
	Technical requirements
	Exploring the Bulk Synchronous Parallel model
	Building a graph processing system in Go
		Queueing and delivering messages
			The Message interface
			Queues and message iterators
			Implementing an in-memory, thread-safe queue
		Modeling the vertices and edges of graphs
			Defining the Vertex and Edge types
			Inserting vertices and edges into the graph
		Sharing global graph state through data aggregation
			Defining the Aggregator interface
			Registering and looking up aggregators
			Implementing a lock-free accumulator for float64 values
		Sending and receiving messages
		Implementing graph-based algorithms using compute functions
			Achieving vertical scaling by executing compute functions in parallel
			Orchestrating the execution of super-steps
		Creating and managing Graph instances
	Solving interesting graph problems
		Searching graphs for the shortest path
			The sequential Dijkstra algorithm
			Leveraging a gossip protocol to run Dijkstra in parallel
		Graph coloring
			A sequential greedy algorithm for coloring undirected graphs
			Exploiting parallelism for undirected graph coloring
		Calculating PageRank scores
			The model of the random surfer
			An iterative approach to PageRank score calculation
			Reaching convergence – when should we stop iterating?
			Web graphs in the real world – dealing with dead ends
			Defining an API for the PageRank calculator
			Implementing a compute function to calculate PageRank scores
	Summary
	Further reading
Chapter 9: Communicating with the Outside World
	Technical requirements
	Designing robust, secure, and backward-compatible REST APIs
		Using human-readable paths for RESTful resources
		Controlling access to API endpoints
			Basic HTTP authentication
			Securing TLS connections from eavesdropping
			Authenticating to external service providers using OAuth2
		Dealing with API versions
			Including the API version as a route prefix
			Negotiating API versions via HTTP Accept headers
		Building RESTful APIs in Go
	Building RPC-based APIs with the help of gRPC
		Comparing gRPC to REST
		Defining messages using protocol buffers
			Defining messages
			Versioning message definitions
			Representing collections
			Modeling field unions
			The Any type
		Implementing RPC services
			Unary RPCs
			Server-streaming RPCs
			Client-streaming RPCs
			Bi-directional streaming RPCs
		Security considerations for gRPC APIs
	Decoupling Links 'R' Us components from the underlying data stores
		Defining RPCs for accessing a remote link-graph instance
		Defining RPCs for accessing a text-indexer instance
		Creating high-level clients for accessing data stores over gRPC 
	Summary
	Questions
	Further reading
Chapter 10: Building, Packaging, and Deploying Software
	Technical requirements
	Building and packaging Go services using Docker
		Benefits of containerization
		Best practices for dockerizing Go applications
		Selecting a suitable base container for your application
	A gentle introduction to Kubernetes
		Peeking under the hood
		Summarizing the most common Kubernetes resource types
		Running a Kubernetes cluster on your laptop!
	Building and deploying a monolithic version of Links 'R' Us
		Distributing computation across application instances
			Carving the UUID space into non-overlapping partitions
			Assigning a partition range to each pod
		Building wrappers for the application services
		The crawler service
		The PageRank calculator service
		Serving a fully functioning frontend to users
			Specifying the endpoints for the frontend application
			Performing searches and paginating results
			Generating convincing summaries for search results
			Highlighting search keywords
		Orchestrating the execution of individual services
		Putting everything together
			Terminating the application in a clean way
			Dockerizing and starting a single instance of the monolith
		Deploying and scaling the monolith on Kubernetes
			Setting up the required namespaces
			Deploying CockroachDB and Elasticsearch using Helm
			Deploying Links 'R' Us
	Summary
	Questions
	Further reading
Section 4: Scaling Out to Handle a Growing Number of Users
Chapter 11: Splitting Monoliths into Microservices
	Technical requirements
	Monoliths versus service-oriented architectures
		Is there something inherently wrong with monoliths?
		Microservice anti-patterns and how to deal with them
	Monitoring the state of your microservices
		Tracing requests through distributed systems
			The OpenTracing project
			Stepping through a distributed tracing example
				The provider service
				The aggregator service
				The gateway
				Putting it all together
			Capturing and visualizing traces using Jaeger
		Making logging your trusted ally
			Logging best practices
			The devil is in the (logging) details
			Shipping and indexing logs inside Kubernetes
				Running a log collector on each Kubernetes node
				Using a sidecar container to collect logs
				Shipping logs directly from the application
		Introspecting live Go services
	Building a microservice-based version of Links 'R' Us
		Decoupling access to the data stores
		Breaking down the monolith into distinct services
		Deploying the microservices that comprise the Links 'R' Us project
			Deploying the link-graph and text-indexer API services
			Deploying the web crawler
			Deploying the PageRank service
			Deploying the frontend service
		Locking down access to our Kubernetes cluster using network policies
	Summary
	Questions
	Further reading
Chapter 12: Building Distributed Graph-Processing Systems
	Technical requirements
	Introducing the master/worker model
		Ensuring that masters are highly available
			The leader-follower configuration
			The multi-master configuration
		Strategies for discovering nodes
		Recovering from errors
	Out-of-core distributed graph processing
		Describing the system architecture, requirements, and limitations
		Modeling a state machine for executing graph computations
		Establishing a communication protocol between workers and masters
			Defining a job queue RPC service
			Establishing protocol buffer definitions for worker payloads
			Establishing protocol buffer definitions for master payloads
		Defining abstractions for working with bi-directional gRPC streams
			Remote worker stream
			Remote master stream
		Creating a distributed barrier for the graph execution steps
			Implementing a step barrier for individual workers
			Implementing a step barrier for the master
		Creating custom executor factories for wrapping existing graph instances
			The workers' executor factory
			The master's executor factory
		Coordinating the execution of a graph job
			Simplifying end user interactions with the dbspgraph package
			The worker job coordinator
				Running a new job
				Transitioning through the stages of the graph's state machine
				Handling incoming payloads from the master
				Using the master as an outgoing message relay
			The master job coordinator
				Running a new job
				Transitioning through the stages for the graph's state machine
				Handling incoming worker payloads
				Relaying messages between workers
		Defining package-level APIs for working with master and worker nodes
			Instantiating and operating worker nodes
			Instantiating and operating master nodes
				Handling incoming gRPC connections
				Running a new job
	Deploying a distributed version of the Links 'R' Us PageRank calculator
		Retrofitting master and worker capabilities to the PageRank calculator service
			Serializing PageRank messages and aggregator values
			Defining job runners for the master and the worker
				Implementing the job runner for master nodes
				The worker job runner
			Deploying the final Links 'R' Us version to Kubernetes
	Summary
	Questions
	Further reading
Chapter 13: Metrics Collection and Visualization
	Technical requirements
	Monitoring from the perspective of a site reliability engineer
		Service-level indicators (SLIs)
		Service-level objectives (SLOs)
		Service-level agreements (SLAs)
	Exploring options for collecting and aggregating metrics
		Comparing push versus pull systems
		Capturing metrics using Prometheus
			Supported metric types
			Automating the detection of scrape targets
				Static and file-based scrape target configuration
				Querying the underlying cloud provider
				Leveraging the API exposed by Kubernetes
			Instrumenting Go code
				Registering metrics with Prometheus
				Vector-based metrics
				Exporting metrics for scraping
	Visualizing collected metrics using Grafana
	Using Prometheus as an end-to-end solution for alerting
		Using Prometheus as a source for alert events
		Handling alert events
			Grouping alerts together
			Selectively muting alerts
			Configuring alert receivers
			Routing alerts to receivers
	Summary
	Questions
	Further reading
Chapter 14: Epilogue
Appendix: Assessments
Other Books You May Enjoy
Index