Reverse Engineering for Beginners

Code Patterns
	The method
	Some basics
		A short introduction to the CPU
		Numeral Systems
		Converting From One Radix To Another
	An Empty Function
		x86
		ARM
		MIPS
		Empty Functions in Practice
	Returning Values
		x86
		ARM
		MIPS
	Hello, world!
		x86
		x86-64
		ARM
		MIPS
		Conclusion
		Exercises
	Function prologue and epilogue
		Recursion
	An Empty Function: redux
	Returning Values: redux
	Stack
		Why does the stack grow backwards?
		What is the stack used for?
		A typical stack layout
		Noise in stack
		Exercises
	Almost empty function
	printf() with several arguments
		x86
		ARM
		MIPS
		Conclusion
		By the way
	scanf()
		Simple example
		The classic mistake
		Global variables
		scanf()
		Exercise
	Worth noting: global vs. local variables
	Accessing passed arguments
		x86
		x64
		ARM
		MIPS
	More about results returning
		Attempt to use the result of a function returning void
		What if we do not use the function result?
		Returning a structure
	Pointers
		Returning values
		Swap input values
	GOTO operator
		Dead code
		Exercise
	Conditional jumps
		Simple example
		Calculating absolute value
		Ternary conditional operator
		Getting minimal and maximal values
		Conclusion
		Exercise
	Software cracking
	Impossible shutdown practical joke (Windows 7)
	switch()/case/default
		Small number of cases
		A lot of cases
		When there are several case statements in one block
		Fall-through
		Exercises
	Loops
		Simple example
		Memory blocks copying routine
		Condition check
		Conclusion
		Exercises
	More about strings
		strlen()
		Boundaries of strings
	Replacing arithmetic instructions to other ones
		Multiplication
		Division
		Exercise
	Floating-point unit
		IEEE 754
		x86
		ARM, MIPS, x86/x64 SIMD
		C/C++
		Simple example
		Passing floating point numbers via arguments
		Comparison example
		Some constants
		Copying
		Stack, calculators and reverse Polish notation
		80 bits?
		x64
		Exercises
	Arrays
		Simple example
		Buffer overflow
		Buffer overflow protection methods
		One more word about arrays
		Array of pointers to strings
		Multidimensional arrays
		Pack of strings as a two-dimensional array
		Conclusion
		Exercises
	Example: a bug in Angband
	Manipulating specific bit(s)
		Specific bit checking
		Setting and clearing specific bits
		Shifts
		Setting and clearing specific bits: FPU example
		Counting bits set to 1
		Conclusion
		Exercises
	Linear congruential generator
		x86
		x64
		32-bit ARM
		MIPS
		Thread-safe version of the example
	Structures
		MSVC: SYSTEMTIME example
		Let\'s allocate space for a structure using malloc()
		UNIX: struct tm
		Fields packing in structure
		Nested structures
		Bit fields in a structure
		Exercises
	The classic struct bug
	Unions
		Pseudo-random number generator example
		Calculating machine epsilon
		FSCALE instruction replacement
		Fast square root calculation
	Pointers to functions
		MSVC
		GCC
		Danger of pointers to functions
	64-bit values in 32-bit environment
		Returning of 64-bit value
		Arguments passing, addition, subtraction
		Multiplication, division
		Shifting right
		Converting 32-bit value into 64-bit one
	LARGE_INTEGER structure case
	SIMD
		Vectorization
		SIMD strlen() implementation
	64 bits
		x86-64
		ARM
		Float point numbers
		64-bit architecture criticism
	Working with floating point numbers using SIMD
		Simple example
		Passing floating point number via arguments
		Comparison example
		Calculating machine epsilon: x64 and SIMD
		Pseudo-random number generator example revisited
		Summary
	ARM-specific details
		Number sign (#) before number
		Addressing modes
		Loading a constant into a register
		Relocs in ARM64
	MIPS-specific details
		Loading a 32-bit constant into register
		Further reading about MIPS
Important fundamentals
	Integral datatypes
		Bit
		Nibble AKA nybble
		Byte
		Wide char
		Signed integer vs unsigned
		Word
		Address register
		Numbers
	Signed number representations
		Using IMUL over MUL
		Couple of additions about two\'s complement form
		-1
	Integer overflow
	AND
		Checking if a value is on 2n boundary
		KOI-8R Cyrillic encoding
	AND and OR as subtraction and addition
		ZX Spectrum ROM text strings
	XOR (exclusive OR)
		Logical difference
		Everyday speech
		Encryption
		RAID4
		XOR swap algorithm
		XOR linked list
		Switching value trick
		Zobrist hashing / tabulation hashing
		By the way
		AND/OR/XOR as MOV
	Population count
	Endianness
		Big-endian
		Little-endian
		Example
		Bi-endian
		Converting data
	Memory
	CPU
		Branch predictors
		Data dependencies
	Hash functions
		How do one-way functions work?
Slightly more advanced examples
	Zero register
	Double negation
	const correctness
		Overlapping const strings
	strstr() example
	qsort() revisited
	Temperature converting
		Integer values
		Floating-point values
	Fibonacci numbers
		Example #1
		Example #2
		Summary
	CRC32 calculation example
	Network address calculation example
		calc_network_address()
		form_IP()
		print_as_IP()
		form_netmask() and set_bit()
		Summary
	Loops: several iterators
		Three iterators
		Two iterators
		Intel C++ 2011 case
	Duff\'s device
		Should one use unrolled loops?
	Division using multiplication
		x86
		How it works
		ARM
		MIPS
		Exercise
	String to number conversion (atoi())
		Simple example
		A slightly advanced example
		Exercise
	Inline functions
		Strings and memory functions
	C99 restrict
	Branchless abs() function
		Optimizing GCC 4.9.1 x64
		Optimizing GCC 4.9 ARM64
	Variadic functions
		Computing arithmetic mean
		vprintf() function case
		Pin case
		Format string exploit
	Strings trimming
		x64: Optimizing MSVC 2013
		x64: Non-optimizing GCC 4.9.1
		x64: Optimizing GCC 4.9.1
		ARM64: Non-optimizing GCC (Linaro) 4.9
		ARM64: Optimizing GCC (Linaro) 4.9
		ARM: Optimizing Keil 6/2013 (ARM mode)
		ARM: Optimizing Keil 6/2013 (Thumb mode)
		MIPS
	toupper() function
		x64
		ARM
		Using bit operations
		Summary
	Obfuscation
		Text strings
		Executable code
		Virtual machine / pseudo-code
		Other things to mention
		Exercise
	C++
		Classes
		ostream
		References
		STL
		Memory
	Negative array indices
		Addressing string from the end
		Addressing some kind of block from the end
		Arrays started at 1
	More about pointers
		Working with addresses instead of pointers
		Passing values as pointers; tagged unions
		Pointers abuse in Windows kernel
		Null pointers
		Array as function argument
		Pointer to a function
		Pointer to a function: copy protection
		Pointer to a function: a common bug (or typo)
		Pointer as object identificator
		Oracle RDBMS and a simple garbage collector for C/C++
	Loop optimizations
		Weird loop optimization
		Another loop optimization
	More about structures
		Sometimes a C structure can be used instead of array
		Unsized array in C structure
		Version of C structure
		High-score file in \"Block out\" game and primitive serialization
	memmove() and memcpy()
		Anti-debugging trick
	setjmp/longjmp
	Other weird stack hacks
		Accessing arguments/local variables of caller
		Returning string
	OpenMP
		MSVC
		GCC
	Signed division using shifts
	Another heisenbug
	The case of forgotten return
	Homework: more about function pointers and unions
	Windows 16-bit
		Example#1
		Example #2
		Example #3
		Example #4
		Example #5
		Example #6
Java
	Java
		Introduction
		Returning a value
		Simple calculating functions
		JVM memory model
		Simple function calling
		Calling beep()
		Linear congruential PRNG
		Conditional jumps
		Passing arguments
		Bitfields
		Loops
		switch()
		Arrays
		Strings
		Exceptions
		Classes
		Simple patching
		Summary
Finding important/interesting stuff in the code
	Identification of executable files
		Microsoft Visual C++
		GCC
		Intel Fortran
		Watcom, OpenWatcom
		Borland
		Other known DLLs
	Communication with outer world (function level)
	Communication with the outer world (win32)
		Often used functions in the Windows API
		Extending trial period
		Removing nag dialog box
		tracer: Intercepting all functions in specific module
	Strings
		Text strings
		Finding strings in binary
		Error/debug messages
		Suspicious magic strings
	Calls to assert()
	Constants
		Magic numbers
		Specific constants
		Searching for constants
	Finding the right instructions
	Suspicious code patterns
		XOR instructions
		Hand-written assembly code
	Using magic numbers while tracing
	Loops
		Some binary file patterns
		Memory \"snapshots\" comparing
	ISA detection
		Incorrectly disassembled code
		Correctly disassembled code
	Other things
		General idea
		Order of functions in binary code
		Tiny functions
		C++
		Crash on purpose
OS-specific
	Arguments passing methods (calling conventions)
		cdecl
		stdcall
		fastcall
		thiscall
		x86-64
		Return values of float and double type
		Modifying arguments
		Taking a pointer to function argument
		Python ctypes problem (x86 assembly homework)
		Cdecl example: a DLL
	Thread Local Storage
		Linear congruential generator revisited
	System calls (syscall-s)
		Linux
		Windows
	Linux
		Position-independent code
		LD_PRELOAD hack in Linux
	Windows NT
		CRT (win32)
		Win32 PE
		Windows SEH
		Windows NT: Critical section
Tools
	Binary analysis
		Disassemblers
		Decompilers
		Patch comparison/diffing
	Live analysis
		Debuggers
		Library calls tracing
		System calls tracing
		Network sniffing
		Sysinternals
		Valgrind
		Emulators
	Other tools
		SMT solvers
		Calculators
	Do You Think Something Is Missing Here?
Case studies
	Mahjong solitaire prank (Windows 7)
	Task manager practical joke (Windows Vista)
		Using LEA to load values
	Color Lines game practical joke
	Minesweeper (Windows XP)
		Finding grid automatically
		Exercises
	Hacking Windows clock
	(Windows 7) Solitaire: practical jokes
		51 cards
		53 cards
	FreeCell prank (Windows 7)
		Part I
		Part II: breaking the Select Game submenu
	Dongles
		Example #1: MacOS Classic and PowerPC
		Example #2: SCO OpenServer
		Example #3: MS-DOS
	Encrypted database case #1
		Base64 and entropy
		Is data compressed?
		Is data encrypted?
		CryptoPP
		Cipher Feedback mode
		Initializing Vector
		Structure of the buffer
		Noise at the end
		Conclusion
		Post Scriptum: brute-forcing IV
	Overclocking Cointerra Bitcoin miner
	Breaking simple executable code encryptor
		Other ideas to consider
	SAP
		About SAP client network traffic compression
		SAP 6.0 password checking functions
	Oracle RDBMS
		V$VERSION table in the Oracle RDBMS
		X$KSMLRU table in Oracle RDBMS
		V$TIMER table in Oracle RDBMS
	Handwritten assembly code
		EICAR test file
	Demos
		10 PRINT CHR$(205.5+RND(1)); : GOTO 10
		Mandelbrot set
	A nasty bug in MSVCRT.DLL
	Other examples
Examples of reversing proprietary file formats
	Primitive XOR-encryption
		Simplest ever XOR encryption
		Norton Guide: simplest possible 1-byte XOR encryption
		Simplest possible 4-byte XOR encryption
		Simple encryption using XOR mask
		Simple encryption using XOR mask, case II
		Homework
	Information entropy
		Analyzing entropy in Mathematica
		Conclusion
		Tools
		A word about primitive encryption like XORing
		More about entropy of executable code
		PRNG
		More examples
		Entropy of various files
		Making lower level of entropy
	Millenium game save file
	fortune program indexing file
		Hacking
		The files
	Oracle RDBMS: .SYM-files
	Oracle RDBMS: .MSB-files
		Summary
	Exercises
	Further reading
Dynamic binary instrumentation
	Using PIN DBI for XOR interception
	Cracking Minesweeper with PIN
		Intercepting all rand() calls
		Replacing rand() calls with our function
		Peeking into placement of mines
		Exercise
	Building Pin
	Why ``instrumentation\'\'?
Other things
	Executable files patching
		x86 code
	Function arguments number statistics
	Compiler intrinsic
	Compiler\'s anomalies
		Oracle RDBMS 11.2 and Intel C++ 10.1
		MSVC 6.0
		ftol2() in MSVC 2012
		Summary
	Itanium
	8086 memory model
	Basic blocks reordering
		Profile-guided optimization
	My experience with Hex-Rays 2.2.0
		Bugs
		Odd peculiarities
		Silence
		Comma
		Data types
		Long and messed expressions
		De Morgan\'s laws and decompilation
		My plan
		Summary
	Cyclomatic complexity
Books/blogs worth reading
	Books and other materials
		Reverse Engineering
		Windows
		C/C++
		x86 / x86-64
		ARM
		Assembly language
		Java
		UNIX
		Programming in general
		Cryptography
		Something even easier
Communities
Afterword
	Questions?
Appendix
	x86
		Terminology
		General purpose registers
		FPU registers
		SIMD registers
		Debugging registers
		Instructions
		npad
	ARM
		Terminology
		Versions
		32-bit ARM (AArch32)
		64-bit ARM (AArch64)
		Instructions
	MIPS
		Registers
		Instructions
	Some GCC library functions
	Some MSVC library functions
	Cheatsheets
		IDA
		OllyDbg
		MSVC
		GCC
		GDB
Acronyms Used
Glossary
Index