Smart Log Data Analytics: Techniques for Advanced Security Analysis

Preface
Acknowledgments
Contents
About the Authors
Acronyms
1 Introduction
	1.1 State of the Art in Security Monitoring and Anomaly Detection
	1.2 Current Trends and …
	1.3 … Future Challenges
	1.4 Log Data Analysis: Today and Tomorrow
	1.5 Smart Log Data Analytics: Structure of the Book
	1.6 Try It Out: Hands-on Examples Throughout the Book
2 Survey on Log Clustering Approaches
	2.1 Introduction
	2.2 Survey Background
		2.2.1 The Nature of Log Data
		2.2.2 Static Clustering
		2.2.3 Dynamic Clustering
		2.2.4 Applications in the Security Domain
	2.3 Survey Method
		2.3.1 Set of Criteria
		2.3.2 Literature Search
	2.4 Survey Results
		2.4.1 Purpose and Applicability (P)
		2.4.2 Clustering Techniques (C)
			2.4.2.1 Types of Static Clustering Techniques
			2.4.2.2 Types of Dynamic Clustering Techniques
			2.4.2.3 Applicability in Live Systems
			2.4.2.4 Non-functional Requirements
		2.4.3 Anomaly Detection (AD)
			2.4.3.1 Static Outlier Detection
			2.4.3.2 Dynamic Anomaly Detection
			2.4.3.3 Cyber Attack Detection
		2.4.4 Evaluation (E)
			2.4.4.1 Evaluation Techniques
			2.4.4.2 Evaluation of Non-functional Requirements
			2.4.4.3 Comparisons and Reproducibility
		2.4.5 Discussion
			2.4.5.1 Problem Domains
			2.4.5.2 Techniques
			2.4.5.3 Benchmarking and Evaluation
	2.5 Conclusion
3 Incremental Log Data Clustering for Processing Large Amounts of Data Online
	3.1 Introduction
	3.2 Concept for Incremental Clustering
		3.2.1 Incremental Clustering
		3.2.2 Description of Model MI
		3.2.3 String Metrics
			3.2.3.1 Levenshtein
			3.2.3.2 Jaro
			3.2.3.3 Sorensen-Dice
			3.2.3.4 Needleman-Wunsch
			3.2.3.5 Longest Common Subsequence
		3.2.4 Description of Model MII
		3.2.5 Time Series Analysis
	3.3 Outlook and Further Development
	3.4 Try It Out
		3.4.1 Exim Mainlog
		3.4.2 Messages Log File
4 Generating Character-Based Templates for Log Data
	4.1 Introduction
	4.2 Concept for Generating Character-Based Templates
	4.3 Cluster Template Generator Algorithms
		4.3.1 Initial Matching
		4.3.2 Merge Algorithm
		4.3.3 Length Algorithm
		4.3.4 Equalmerge Algorithm
		4.3.5 Token_char Algorithm
		4.3.6 Comparison
	4.4 Outlook and Further Development
	4.5 Try it Out
		4.5.1 Exim Mainlog
5 Time Series Analysis for Temporal Anomaly Detection
	5.1 Introduction
	5.2 Concept for Dynamic Clustering and AD
	5.3 Cluster Evolution
		5.3.1 Clustering Model
		5.3.2 Tracking
		5.3.3 Transitions
		5.3.4 Evolution Metrics
	5.4 Time Series Analysis
		5.4.1 Model
		5.4.2 Forecast
		5.4.3 Correlation
		5.4.4 Detection
	5.5 Example
		5.5.1 Long-Term Analysis of Suricata Logs
		5.5.2 Short-Term Analysis of Audit Logs
6 AECID: A Light-Weight Log Analysis Approach for Online Anomaly Detection
	6.1 Introduction
	6.2 The AECID Approach
		6.2.1 AMiner
		6.2.2 AECID Central
		6.2.3 Detecting Anomalies
		6.2.4 Rule Generator
		6.2.5 Correlation Engine
		6.2.6 Detectable Anomalies
	6.3 System Deployment and Operation
	6.4 Application Scenarios
	6.5 Try It Out
		6.5.1 Configuration of the AMiner for AIT-LDSv1.1
		6.5.2 Apache Access Logs
		6.5.3 Exim Mainlog File
		6.5.4 Audit Logs
7 A Concept for a Tree-Based Log Parser Generator
	7.1 Introduction
	7.2 Tree-Based Parser Concept
	7.3 AECID-PG: Tree-Based Log Parser Generator
		7.3.1 Challenges When Generating Tree-Like Parsers
		7.3.2 AECID-PG Concept
		7.3.3 AECID-PG Rules
		7.3.4 Features
	7.4 Outlook and Further Application
	7.5 Try it Out
		7.5.1 Exim Mainlog
		7.5.2 Audit Logs
8 Variable Type Detector for Statistical Analysis of Log Tokens
	8.1 Introduction
	8.2 Variable Type Detector Concept
	8.3 Variable Type Detector Algorithm
		8.3.1 Sanitize Log Data
		8.3.2 Initialize Types
		8.3.3 Update Types
		8.3.4 Compute Indicators
			8.3.4.1 Token Indicator
			8.3.4.2 Event Indicator
		8.3.5 Select Tokens
		8.3.6 Compute Indicator Weights
		8.3.7 Report Anomalies
	8.4 Try It Out
		8.4.1 Apache Access Log
9 Final Remarks
A Getting Started with AIT\'s AMiner
	A.1 Requirements
	A.2 Installation
	A.3 First Very Simple Configuration
	A.4 Detecting Anomalies in Combinations of Different Log Line Fields
B Description of the AIT Log Data Set (AIT-LDSv1.1)
	B.1 Testbed Design
	B.2 Log Files
	B.3 Attacks
C Going Further: Integrating AMiner with SIEM Solutions
	C.1 Interfaces
	C.2 ELK Stack
	C.3 QRadar
References