Reinforcement Learning for Cyber Operations

Introduction
Chapter 1 Motivation
	1.1 Introduction
		1.1.1 Cyberattack Campaigns via MITRE ATT&CK
	1.2 Attack Graphs
	1.3 Cyber Terrain
	1.4 Penetration Testing
	1.5 AI Reinforcement Learning Overview
	1.6 Organization of the Book
	References
Chapter 2 Overview of Penetration Testing
	2.1 Penetration Testing
		2.1.1 Introduction to Red Teaming
			2.1.1.1 Why? Reasons for Red Team Penetration Testing
			2.1.1.2 Teamwork: Red–Blue–Purple Teaming
		2.1.2 A Brief History of Red Teams What and Where
			2.1.2.1 Military, Government, and Defense Industry
			2.1.2.2 Financial Services and Commerce
			2.1.2.3 Healthcare
			2.1.2.4 Technology, Telecommunications, and Cyber
			2.1.2.5 Conclusion
		2.1.3 Modern Penetration Testing
			2.1.3.1 Types and Styles of Pentesting Engagements
			2.1.3.2 Black Box
			2.1.3.3 Gray Box
			2.1.3.4 White Box
		2.1.4 Objectives, Considerations, and Goals During a Penetration Test
			2.1.4.1 Objectives
			2.1.4.2 Considerations
		2.1.5 Methodology
			2.1.5.1 Thinking Like an Adversary
			2.1.5.2 The Hacker Mindset
			2.1.5.3 Pentesting Phases
			2.1.5.4 Planning and Reconnaissance Phase (aka Information Gathering)
			2.1.5.5 Scanning
			2.1.5.6 Port Scanners
			2.1.5.7 Network Discovery Tools
			2.1.5.8 Vulnerabilities Assessment Phase
			2.1.5.9 Exploitation Phase
			2.1.5.10 Post‐exploitation (Reassessment) Phase
			2.1.5.11 Reporting
	2.2 Importance of Data
		2.2.1 Data Types, Data Sources, and Pivot Points
			2.2.1.1 Scanning Data via Port Scanners and Network Scanners
			2.2.1.2 Application Identifications (Banners)
			2.2.1.3 Operating System Identification
			2.2.1.4 Network Topology
			2.2.1.5 Network Scanners
			2.2.1.6 External “Passive” Data Sources
			2.2.1.7 Databases
			2.2.1.8 Vulnerabilities Databases: CVE and CVSS Databases
			2.2.1.9 Data Formats
			2.2.1.10 Nessus Typical File/Terminal Output
			2.2.1.11 Nessus Output, CSV Format
			2.2.1.12 Nessus Output, XML Format
			2.2.1.13 OpenVAS Standard Format
			2.2.1.14 OpenVAS.csv Format
	2.3 Conclusion
	References
Chapter 3 Reinforcement Learning: Theory and Application
	3.1 An Introduction to Reinforcement Learning (RL)
	3.2 RL and Markov Decision Processes
	3.3 Learnable Functions for Agents
		3.3.1 The Policy Model
		3.3.2 The Value‐Based Model
		3.3.3 Model‐Based Learning
		3.3.4 Combining Methods
	3.4 Enter Deep Learning
	3.5 Q‐Learning and Deep Q‐Learning
		3.5.1 Boltzmann Policies and Experience Replay
		3.5.2 Implementing DQN
			3.5.2.1 The CartPole Environment
			3.5.2.2 DQN Architecture
			3.5.2.3 Boltzmann Policy for Action Selection
			3.5.2.4 Experience Replay Mechanism
			3.5.2.5 The Training Process
			3.5.2.6 Post‐Training
	3.6 Advantage Actor‐Critic (A2C)
		3.6.1 The Actor
		3.6.2 The Critic and Advantage
		3.6.3 Implementing A2C
			3.6.3.1 Actor and Critic Networks
			3.6.3.2 The GAE Function
			3.6.3.3 Training the Model
	3.7 Proximal Policy Optimization
		3.7.1 Trust Region Policy Optimization (TRPO)
		3.7.2 Proximal Policy Optimization (PPO)
			3.7.2.1 PPO with KL Penalties
			3.7.2.2 PPO with Clipped Objectives
	3.8 Conclusion
	References
Chapter 4 Motivation for Model‐driven Penetration Testing
	4.1 Introduction
	4.2 Limits of Modern Attack Graphs
		4.2.1 Critiques of MDPs with Attack Graphs
		4.2.2 Ontology‐based Approaches
	4.3 RL for Penetration Testing
	4.4 Modeling MDPs
		4.4.1 Whole Campaign Emulation
	4.5 Conclusion
	References
Chapter 5 Operationalizing RL for Cyber Operations
	5.1 A High‐Level Architecture
	5.2 Layered Reference Model
		5.2.1 Real Network Processes
		5.2.2 Attack Graph Generation Processes
		5.2.3 MDP Construction Processes
		5.2.4 Machine Learning Processes
		5.2.5 LRM‐RAG Review
		5.2.6 LRM‐RAG Limitations
	5.3 Key Challenges for Operationalizing RL
		5.3.1 Generation and Actuation
		5.3.2 Realism
		5.3.3 Unstable and Evolving Networks
	5.4 Conclusions
	References
Chapter 6 Toward Practical RL for Pen‐Testing
	6.1 Current Challenges to Practicality
		6.1.1 The Problem of Scaling
		6.1.2 The Problem of Realism
	6.2 Practical Scalability in RL
		6.2.1 State and Action Spaces
		6.2.2 A Flavor of Double Agent
		6.2.3 The Workhorse: Double Agent + PPO (DA‐PPO)
	6.3 Model Realism
		6.3.1 Reward Engineering
		6.3.2 Human Inputs vs. Model Inputs
	6.4 Examples of Applications
		6.4.1 SDR
		6.4.2 Crown Jewels Analysis
		6.4.3 Discovering Exfiltration Paths with RL
		6.4.4 C2
		6.4.5 Ransomware
	6.5 Realism and Scale
		6.5.1 Multi‐task Learning
		6.5.2 Multi‐Objective Learning
	References
Chapter 7 Putting it Into Practice: RL for Scalable Penetration Testing
	7.1 Crown Jewels Analysis
		7.1.1 Overview and Motivation
		7.1.2 Network Setup for Evaluation
		7.1.3 Reward Calculation
		7.1.4 Model Architecture
		7.1.5 Training Process
		7.1.6 Experimental Results
	7.2 Discovering Exfiltration Paths
		7.2.1 Overview and Motivation
		7.2.2 Network Setup for Evaluation
		7.2.3 Reward Calculation
		7.2.4 Model Architecture
		7.2.5 Experimental Results
	7.3 Discovering Command and Control Channels
		7.3.1 Overview and Motivation
		7.3.2 Network Setup for Evaluation
			7.3.2.1 Three‐Stage C2 Attack Model
			7.3.2.2 Network Exploration and Exploitation
			7.3.2.3 Connection and Exfiltration Phases
			7.3.2.4 Firewall Dynamics
			7.3.2.5 RL Formulation ‐ State Space, Action Space, and Reward Function
		7.3.3 Model Architecture and Training
			7.3.3.1 Training Methodology and Network Architecture
			7.3.3.2 Hyperparameters
			7.3.3.3 Training Execution and Computational Resources
		7.3.4 Experimental Results
			7.3.4.1 Training and Convergence
			7.3.4.2 Evaluation of Learned Policy
			7.3.4.3 Behavioral Analysis of RL Agent
			7.3.4.4 Avoidance of Firewall Detection
	7.4 Exposing Surveillance Detection Routes
		7.4.1 Overview and Motivation
		7.4.2 Network Setup for Evaluation
		7.4.3 The Warm‐Up Phase
		7.4.4 Model Architectures and Training
		7.4.5 Experimental Results
	7.5 Enhanced Exfiltration Path Analysis
		7.5.1 Overview and Motivation
		7.5.2 Network Setup for Evaluation
			7.5.2.1 Exfiltration Campaign Model
			7.5.2.2 Network Firewalls and Monitoring
			7.5.2.3 Protocol‐Based Path Selection
			7.5.2.4 Action Clock Time and Reward Function
			7.5.2.5 Training and Evaluation Networks
			7.5.2.6 State and Action Spaces
		7.5.3 Model Architecture and Training
			7.5.3.1 Model Architecture and Training Approach
			7.5.3.2 Hyperparameters in Training
			7.5.3.3 Training Episodes and Payload Targets
		7.5.4 Experimental Results
			7.5.4.1 Performance and Convergence
			7.5.4.2 Attack Path Analysis
			7.5.4.3 Strategic Actions and Protocol Utilization
	References
Chapter 8 Using and Extending These Models
	8.1 Supplementing Penetration Testing
		8.1.1 Vulnerability Discovery
		8.1.2 Path Analysis
	8.2 Risk Scoring
		8.2.1 Current State
		8.2.2 Future State
	8.3 Further Modeling
		8.3.1 Simulation and Threat Detection
		8.3.2 Ransomware Detection
		8.3.3 Engineering New Exploits
		8.3.4 Extension to LLMs
		8.3.5 Asset Discovery and Classification
		8.3.6 Attribution
			8.3.6.1 Detecting Malicious Behavior
			8.3.6.2 Attributing with Attack Paths
		8.3.7 Defensive Modeling
		8.3.8 AI vs. AI
	8.4 Generalization
		8.4.1 Running Live
		8.4.2 Teaching Computers to Attack Computers
		8.4.3 Where the Arms Race is Racing Toward
	References
Chapter 9 Model‐driven Penetration Testing in Practice
	9.1 Recap
		9.1.1 Using Cyber Terrain
		9.1.2 Crown Jewels
		9.1.3 Exfiltration
		9.1.4 Surveillance Detection Routes (SDR): Advanced Reconnaissance
	9.2 The Case for Model‐driven Cyber Detections
		9.2.1 The Environment
		9.2.2 The CVSS Attack Graph
		9.2.3 Layering Defensive Terrain
		9.2.4 The AI Agents
		9.2.5 The Structuring Agent
		9.2.6 The Exploiting Agent
		9.2.7 The Intuition
		9.2.8 The Training Algorithm
		9.2.9 Learning in Simulation vs. Learning in Reality
		9.2.10 Putting it in Practice
			9.2.10.1 The Motivation and Experimental Design
			9.2.10.2 Network Design, Assumptions, and Defensive Terrain
			9.2.10.3 The Warmup
			9.2.10.4 Results
			9.2.10.5 Creating Actionable Intelligence
			9.2.10.6 Attack Surface Characterization (ASC)
			9.2.10.7 Risk Management Considerations
	References
A Appendix
Index