Mastering GitLab 12: Implement DevOps culture and repository management solutions

Cover
Title Page
Copyright and Credits
Dedication
About Packt
Contributors
Table of Contents
Preface
Section 1: Install and Set Up GitLab On-Premises or in the Cloud
Chapter 1: Introducing the GitLab Architecture
	Technical requirements
	The origins of GitLab
	Exploring GitLab editions – CE and EE
	The core system components of GitLab
		NGINX
			Debugging NGINX
		Unicorn
			Debugging Unicorn
				Timeouts in Unicorn logs
				Unicorn processes disappear 
				Other kinds of errors or 100% CPU load
		Sidekiq
			Debugging Sidekiq
				Sidekiq processes disappear
				A Sidekiq process is seemingly doing nothing
				Other kind of errors or 100% CPU load
		GitLab Shell
			Debugging GitLab Shell
		Redis
			Basic data operations in Redis
		Gitaly
			Debugging Gitaly
		GitLab Workhorse
			Debugging GitLab Workhorse
		Database
			Debugging PostgreSQL
	GitLab CI
		Pipelines and jobs
	GitLab Runners
		Issues with the old runner
		Switching to Go
	Cloud native
	Summary
	Questions
	Further reading
Chapter 2: Installing GitLab
	Technical requirements
	Installing GitLab using omnibus packages
		Omnibus structure
			Project definition
			Individual software definitions
			A GitLab configuration template
			Chef components
			Runit recipe
			Tests
			gitlab-ctl commands
		Running the installer
			Browsing to the external URL and login
			Upgrade using the omnibus-gitlab package
	Running from source
		Operating system – Debian 10
			Required basic software packages
			Required programming languages
				Ruby
				Go
				Node.js
			System users
			SQL database
			Redis memory database
			GitLab
			Installing GitLab Shell
			Installing GitLab-Workhorse
			Installing Gitaly
			Initializing the database and activating advanced features
			Final steps for preparing the system
			Preparing to serve
				Compiling GetText PO files
				Compiling assets
				Starting your GitLab instance
			NGINX
	Using it from Docker
		Running the image directly
			Configuring GitLab after startup
			Starting the container with configuration settings as input
			Upgrading GitLab
			Run GitLab CE on a different IP address
			Debugging the container
		Install GitLab using Docker Compose
			Updating GitLab using Docker Compose
	Deploying GitLab using Kubernetes
		GitLab Runner Helm chart
			Deploying of a GitLab Runner to Kubernetes
		GitLab Helm chart
			Deploying GitLab to Kubernetes
			Monitoring the deployment
			Initial login
				Outgoing email
				Updating GitLab using the Helm chart
				Uninstalling GitLab using the Helm chart
		Creating droplets on DigitalOcean
	Summary
	Questions
	Further reading
Chapter 3: Configuring GitLab Using the Web UI
	Technical requirements
	Configuring GitLab settings at the instance level
		Menu options
		Monitoring
		Messages
		System hooks
			Plugins
		Applications
		Abuse reports
		License
		Kubernetes
		Push rules
		Geo
		Deploy Keys
		Service templates
		Appearance
		Settings
			General
				Visibility and access controls
				Account and limit
				Diff limits
				Sign-up restrictions
				Sign-in restrictions
				Terms of service and privacy policy
				External authentication
				Web Terminal
				Web IDE
		Integrations
			Elasticsearch
			PlantUML
			Third-party offers
			Snowplow
		Repository
			Repository mirror
			Repository storage
			Repository maintenance
		Templates
		CI/CD
			Auto DevOps settings
			Shared runner settings
			Container registry
		Reporting
			Spam and anti-bot protection
			Abuse reports
			Error reporting and logging
		Metrics and profiling
			Metrics – InfluxDB
			Metrics – Prometheus
			Profiling – Performance Bar
			Usage statistics
			Pseudonymizer data collection
		Network
			Performance optimization
			User and IP rate limits
			Outbound requests
			Geo
		Preferences
			Email
			Help page
			Pages
			Real-time features
			Gitaly
			Localization
	Configuring GitLab settings at the group level
	Configuring GitLab settings at the project level
		General
			Naming, topics, avatar
			Visibility, project features, permissions
			Merge requests
	Summary
	Questions
	Further reading
Chapter 4: Configuring GitLab from the Terminal
	Technical requirements
	Configuring omnibus and GitLab installations from the terminal
	Configuring source installations
		GitLab app settings
			Storing big files
				Using object storage
		GitLab pages
		Mattermost
		Gravatar
		Sidekiq
			GitLab Registry
		GitLab CI settings
		Auth settings
		Advanced settings
		Rack Attack
	Reconfiguring GitLab Docker containers
	Changing GitLab in a Kubernetes environment
		Basic configuration
			Configuring TLS
			Configuring outgoing emails
			Other settings
	Summary
	Questions
	Further reading
Section 2: Migrating Data from Different Locations
Chapter 5: Importing Your Project from GitHub to GitLab
	Technical requirements
	Using the GitHub integration feature
		Preparing GitHub for export
		Preparing GitLab for import
		Running the import
	Using a GitHub token
		Preparing GitHub for export
		Running the import
	Using a GitLab rake task
		Preparing GitLab for import
		Running the import
	Summary
	Questions
	Further reading
Chapter 6: Migrating from CVS
	Technical requirements
	CVS versus Git
		Filesets versus changesets
		Git branching
		Creating repositories
		Atomic operations
		Object naming or referencing versions
		Keyword substitution
		Binary blobs
		Amending commits
		A central repository
		Accompanying toolset
		Detecting file renames
		Commit before merge
	Preparing to migrate from CVS to Git
		Preparing for a conversion using cvs-fast-export
		Preparing for a conversion using cvs2git
	Running the conversion
		Converting data using cvs-fast-export
		Converting data using cvs2git
	Cleaning up after migration
	Summary
	Questions
	Further reading
Chapter 7: Switching from SVN
	Technical requirements
	The difference between SVN and Git
		Security and access control
		Space requirements and references
		Branching
		Handling binaries with SVN and Git
	Mirroring SVN and GIT
		No sync, just convert
	Using svn2git to migrate in one cut
	Summary
	Questions
	Further reading
Chapter 8: Moving Repositories from TFS
	Technical requirements
	TFS versus Git
	Centralized or decentralized
		Handling changes
			Branching and merging capacity
			History
			Traceability
			File handling
	The git-tfs tool
		Preparing to migrate
	Summary
	Questions
	Further reading
Section 3: Implement the GitLab DevOps Workflow
Chapter 9: GitLab Vision - the Whole Toolchain in One Application
	Technical requirements 
	The Agile Manifesto
		The initial model – waterfall
		Royce\'s model
		The sashimi model
		The V-model
		DSDM
			Timeboxing
			MoSCoW
		Scrum
		Crystal methods
	XP
		Fine-scale feedback
			Planning game
				Release planning
				Iteration planning
			Pair programming
			Test Driven Development
		Continuous processes
			Continuous integration
			Refactoring
			Short iterations
		Everybody owns the code
			Shared understanding
			Coding standards
			Simple design
			System metaphor
	The DevOps movement
		History of the movement
			Four Quadrant Model
			Four levels of competence
	The toolchain
	Summary
	Questions
	Further reading
Chapter 10 Create Your Product, Verify, and Package it
	Technical requirements
	The GitLab workflow
	DevOps phase – manage your ideas
		Cycle analytics
	DevOps phase – plan your feature
		Issues
			Content
			Status of the issue
			Meta information
		Discussions
		Milestones
		Epics
		Time tracking
		Quick actions
		The Project Issue board
		Todos
	DevOps phase – create it 
		Projects and groups
		Snippets
		Web IDE
		Wiki
			Protected branches
		Merge requests
	DevOps phase – verify your product
		Code Quality reports
		Review apps
	DevOps phase – package it for use
		GitLab container registry
	Summary
	Questions
	Further reading
Chapter 11: The Release and Configure Phase
	Technical requirements
	Continuous Deployment
	Auto DevOps
		Configuring Auto DevOps
			Build step
			Code quality scan
			Container scanning
			Dependency scanning
			License management
			Static application security testing (sast)
			The final test step
			Production
			Performance
	Summary
	Questions
	Further reading
Chapter 12: Monitoring with Prometheus
	Technical requirements
	Setting up Prometheus
		Using an external Prometheus host
		Enabling the external dashboard link
	Customizing monitoring
	The static analysis of security vulnerabilities
	Dynamic Application Security Testing
	Dependency checking
	Summary
	Questions
	Further reading
Chapter 13: Integrating GitLab with CI/CD Tools
	Technical requirements
	Using Jira with GitLab 
	Connecting Jenkins to GitLab
	Integrating with Mattermost
	Using webhooks for events
	Summary
	Questions
	Further reading
Section 4: Utilize GitLab CI and CI Runners
Chapter 14: Setting Up Your Project for GitLab Continuous Integration
	Technical requirements
	Pipelines
	Jobs
	Creating .gitlab-ci.yml
	Configuring a runner
		GitLab Runner features
	Summary
	Questions
	Further reading
Chapter 15: Installing and Configuring GitLab Runners
	Technical requirements
	The Runner client architecture
		Basic architecture
	Creating a basic Runner with the shell executor
		Installing Runner on Linux
			Using a package manager
			Using a manual installation
			Updating a manually installed runner binary
		Installing on Mac
			The manual way of installing a runner
			Installing and using the Homebrew package manager
			Updating a manually installed runner binary
		Installing on Windows
		Registering a runner
			The interactive way of registering a runner
			The non-interactive way of registering a runner
		Running the nightly version
	Summary
	Questions
	Further reading
Chapter 16: Using GitLab Runners with Docker or Kubernetes
	Technical requirements
	Runner client architecture
	Creating your own Dockerized GitLab Runner
	Using a prebuilt Docker container to deploy GitLab Runners
	Using a Kubernetes cluster to spawn GitLab Runners
	Summary
	Questions
	Further reading
Chapter 17: Autoscaling GitLab CI Runners
	Technical requirements
	Runner client architecture
	Setting up the environment
		Preparing a bastion host
		Deploying the GitLab Runner software
		Installing Docker Machine
	Configuring the Runner
		Off-peak time mode configuration
		Distributed runners caching
			Setting the cache globally 
			Setting the cache at the project level
		Distributed container registry mirroring
		Installing and running a proxy container registry and a caching server
			Proxy container registry
			Caching server
				Creating an S3 bucket in Amazon Web Services
				Creating your own MinIO server
		Scaling your runners
			Using Docker Machines with a local VirtualBox instance
			Using docker machines that have been created on Amazon Web Services (EC2)
	Summary
	Questions
	Further reading
Chapter 18: Monitoring CI Metrics
	Technical requirements
	Enabling monitoring for Runners
	Editing the GitLab Runner configuration file
	Runner business logic metrics
		Key metrics to watch
	General process metrics
		Key metrics to watch
		Alert management
	Summary
	Questions
	Further reading
Section 5: Scale the Server Infrastructure (High Availability Setup)
Chapter 19: Creating a Basic HA Architecture Using Horizontal Scaling
	Technical requirements
	The underlying architecture of this solution
		Amazon services
			Elastic compute cloud (EC2)
			Classic load balancer
			Virtual private cloud and subnets
			SGs
		Terraform
			Installing Terraform
		Ansible
		Installing the Ansible Terraform provider
		Starting with the code
			vpc.tf
			subnet.tf
			instance.tf
			ansible_host.tf
			route_table.tf
			security_group.tf
			variable.tf
			keypair.tf
			lb.tf
			providers.tf
		Preparing to run Terraform to deploy the virtual hardware
		Running the deployment
	Setting up the bastion hosts
	Configuring the database nodes
		Contents of the gitlab.rb.postgres.j2 template
		Configuring the consul nodes
			Contents of gitlab.rb.consul.j2
		Configuring the PgBouncer node
			Contents of gitlab.rb.pgbouncer.j2
	The Redis configuration
		Contents of gitlab.rb.redis.j2
	Connecting the shared filesystem
		Contents of nfs_exports.j2
	Setting up the application servers
		Contents of gitlab.rb.j2
		Running all the Ansible playbooks
	Summary
	 Questions
	 Further reading
Chapter 20: Managing a Hybrid HA Environment
	Technical requirements
	The basic architecture of this solution
	A renewed Terraform configuration
		instance.tf
		ansible_host.tf
	Splitting application components into frontend and middleware tiers
		Splitting Sidekiq from the frontend
			Contents of gitlab.rb.middleware.j2
		Creating a monitoring instance
			Contents of gitlab.rb.prometheus.j2
		Creating a monitoring dashboard with Grafana
			Contents of gitlab.rb.grafana.j2
	Connecting the shared filesystem
		Contents of the gitlab.rb.gitaly.j2 file
	Changes in Ansible files
	Script enhancements
	Summary
	Questions
	Further reading
Chapter 21: Making Your Environment Fully Distributed
	Technical requirements
	The basic architecture of this solution
	Performing changes to the Terraform configuration
		instance.tf
		ansible_host.tf
	Splitting more application components
		The third application server for Git SSH
			Contents of gitlab.rb.frontend_ssh.j2
		The middleware layer – Sidekiq 
			The ASAP Sidekiq instance
				Contents of gitlab.rb.sidekiq_asap.j2
			The real-time Sidekiq instance
			Contents of gitlab.sidekiq.realtime.j2
			The pipeline Sidekiq instance
			Contents of gitlab.sidekiq.pipeline.j2
			The normal Sidekiq instance
	The clustered Redis/Sentinel configuration
		Contents of gitlab.rb.redis-cluster.j2
	Changes in Ansible files
	Script enhancements
	Summary
	Questions
	Further reading
Chapter 22: Using Geo to Create Distributed Read-Only Copies of GitLab
	Technical requirements
	The basic architecture of this solution
	Preparing the infrastructure
		The root module explained – main.tf
		Utilizing the keypair module – modules/services/keypair/main.tf
		Variables for the keypair module – modules/services/keypair/vars.tf
		Outputs from the keypair module – modules/services/keypair/output.tf
		Explaining the GitLab module
			The main module file – modules/services/gitlab/main.tf
			The variable file – modules/services/gitlab/vars.tf
			The outputs for the module – modules/services/gitlab/outputs.tf
	Setting up Geo
		Installing the GitLab software and license
			Contents of gitlab.rb.j2
		Preparing the database replicas 
			Contents of gitlab.rb.primary.j2
			Contents of gitlab.rb.primary.j2
		Changing the SSH key lookup method to the database
		Adding the secondary node via the web UI
			Activating hashed storage
			Checking the status of the secondary node
	Summary
	Questions
	Further reading
Assessments
Other Books You May Enjoy
Index