Microscope Image Processing

Front Cover
Microscope Image Processing
Copyright
Contents
Foreword to the First Edition
	Reference
Foreword to the Second Edition
Preface to the First Edition
Preface to the Second Edition
Acknowledgments
Chapter One: Introduction
	1.1. The Microscope and Image Processing
	1.2. The Scope of This Book
	1.3. Our Approach
		1.3.1. The Four Types of Images
			1.3.1.1. The Optical Image
			1.3.1.2. The Continuous Image
			1.3.1.3. The Digital Image
			1.3.1.4. The Displayed Image
		1.3.2. The Result
			1.3.2.1. Analytic Functions
		1.3.3. The Sampling Theorem
	1.4. The Challenge
	1.5. Modern Microscopy
	1.6. Nomenclature
	1.7. Summary of Important Points
	References
Chapter Two: Fundamentals of Microscopy
	2.1. The Origins of the Microscope
	2.2. Optical Imaging
		2.2.1. Image Formation by a Lens
			2.2.1.1. Imaging a Point Source
			2.2.1.2. Focal Length
			2.2.1.3. Magnification
			2.2.1.4. Numerical Aperture
			2.2.1.5. Lens Shape
	2.3. Diffraction Limited Optical Systems
		2.3.1. Linear System Analysis
	2.4. Incoherent Illumination
		2.4.1. The Point Spread Function
		2.4.2. The Optical Transfer Function
	2.5. Coherent Illumination
		2.5.1. The Coherent Point Spread Function
		2.5.2. The Coherent Optical Transfer Function
	2.6. Resolution
		2.6.1. The Abbe Distance
		2.6.2. The Rayleigh Distance
		2.6.3. Size Calculations
	2.7. Aberration
	2.8. Calibration
		2.8.1. Spatial Calibration
		2.8.2. Photometric Calibration
	2.9. Summary of Important Points
	References
Chapter Three: Image Digitization and Display
	3.1. Introduction
	3.2. Digitizing Images
		3.2.1. Resolution
		3.2.2. Sampling
		3.2.3. Interpolation
		3.2.4. Aliasing
		3.2.5. Noise
		3.2.6. Shading
		3.2.7. Photometry
		3.2.8. Geometric Distortion
	3.3. Overall System Design
		3.3.1. Cumulative Resolution
		3.3.2. Design Rules of Thumb
			3.3.2.1. Pixel Spacing
			3.3.2.2. Resolution
			3.3.2.3. Noise
			3.3.2.4. Photometry
			3.3.2.5. Distortion
	3.4. Image Display
		3.4.1. Volatile Displays
		3.4.2. Displayed Image Size
		3.4.3. Aspect Ratio
		3.4.4. Photometric Resolution
		3.4.5. Grayscale Linearity
		3.4.6. Low-frequency Response
		3.4.7. High-frequency Response
			3.4.7.1. Sampling for Display Purposes
			3.4.7.2. Oversampling
			3.4.7.3. Resampling
		3.4.8. Noise
	3.5. Summary of Important Points
	References
Chapter Four: Geometric Transformations
	4.1. Introduction
	4.2. Implementation
	4.3. Gray Level Interpolation
		4.3.1. Nearest Neighbor Interpolation
		4.3.2. Bilinear Interpolation
		4.3.3. Bicubic Interpolation
		4.3.4. Higher-order Interpolation
	4.4. The Spatial Transformation
		4.4.1. Control Grid Mapping
	4.5. Applications
		4.5.1. Distortion Removal
		4.5.2. Image Registration
		4.5.3. Stitching
	4.6. Summary of Important Points
	References
Chapter Five: Image Enhancement
	5.1. Introduction
	5.2. Spatial Domain Enhancement Methods
		5.2.1. Contrast Stretching
		5.2.2. Clipping and Thresholding
		5.2.3. Image Subtraction and Averaging
		5.2.4. Histogram Equalization
		5.2.5. Histogram Specification
		5.2.6. Spatial Filtering
		5.2.7. Directional and Steerable Filtering
		5.2.8. Median Filter
		5.2.9. Anisotropic Diffusion Filter
	5.3. Fourier Transform Methods
		5.3.1. Wiener Filtering and Wiener Deconvolution
		5.3.2. Deconvolution Using a Least Squares Approach
		5.3.3. Low-Pass Filtering
		5.3.4. High-pass and Band-pass Filtering
	5.4. Wavelet Transform Methods
		5.4.1. Wavelet Thresholding
		5.4.2. Differential wavelet transform and multiscale pointwise product
	5.5. Color Image Enhancement
		5.5.1. Pseudo-Color Transformations
		5.5.2. Color Image Smoothing
		5.5.3. Color Image Sharpening
	5.6. Summary of Important Points
	References
Chapter Six: Morphological Image Processing
	6.1. Introduction
	6.2. Binary Morphology
		6.2.1. Binary Erosion and Dilation
		6.2.2. Binary Opening and Closing
		6.2.3. Binary Morphological Reconstruction From Markers
			6.2.3.1. Connectivity
			6.2.3.2. Markers
			6.2.3.3. A Priori Selection Using the Image Border for Marker Placement
			6.2.3.4. Reconstruction From Opening
		6.2.4. Reconstruction Using Area Opening and Closing
		6.2.5. Skeletonization
	6.3. Grayscale Operations
		6.3.1. Threshold Sets and Level Sets
		6.3.2. Grayscale Erosion and Dilation
			6.3.2.1. Morphological Gradient
		6.3.3. Grayscale Opening and Closing
			6.3.3.1. The Top-Hat Concept
			6.3.3.2. Grayscale Image Filtering
		6.3.4. Component Filters and Grayscale Morphological Reconstruction
			6.3.4.1. The Reconstruction Process
			6.3.4.2. Grayscale Area Opening and Closing
			6.3.4.3. Edge-Off Operators
			6.3.4.4. h-Maxima and h-Minima Operations
			6.3.4.5. Regional Maxima
			6.3.4.6. Marker Extraction
	6.4. Watershed Segmentation
		6.4.1. The Classical Watershed Transform
		6.4.2. Filtering the Minima
		6.4.3. Texture Detection
		6.4.4. Watershed From Markers
		6.4.5. Segmentation of Overlapped Convex Cells
		6.4.6. Inner and Outer Markers
	6.5. Summary of Important Points
	References
Chapter Seven: Image Segmentation
	7.1. Introduction
		7.1.1. Pixel Connectivity
	7.2. Region-Based Segmentation
		7.2.1. Thresholding
			7.2.1.1. Global Thresholding
			7.2.1.2. Adaptive Thresholding
			7.2.1.3. Threshold Selection
				Histogram Smoothing
				The ISODATA Algorithm
				The Background Symmetry Algorithm
				The Triangle Algorithm
				Gradient-Based Algorithms
			7.2.1.4. Thresholding Circular Spots
			7.2.1.5. Thresholding Noncircular and Noisy Spots
				Noncircular Spots
				Objects of General Shape
		7.2.2. Morphological Processing
			7.2.2.1. Hole Filling
			7.2.2.2. Border Object Removal
			7.2.2.3. Separation of Touching Objects
			7.2.2.4. The Watershed Algorithm
		7.2.3. Region Growing
		7.2.4. Region Splitting
	7.3. Boundary-Based Segmentation
		7.3.1. Boundaries and Edges
		7.3.2. Boundary Tracking Based on Maximum Gradient Magnitude
		7.3.3. Boundary Finding Based on Gradient Image Thresholding
		7.3.4. Boundary Finding Based on Laplacian Image Thresholding
		7.3.5. Boundary Finding Based on Edge Detection and Linking
			7.3.5.1. Edge Detection
				The Roberts Edge Detector
				The Sobel Edge Detector
				The Prewitt edge detector
				The Canny Edge Detector
			7.3.5.2. Edge Linking and Boundary Refinement
				Heuristic Search
				Curve Fitting
				The Hough Transform
				Active Contours
		7.3.6. Encoding Segmented Images
			7.3.6.1. The Object Label Map
			7.3.6.2. The Boundary Chain Code
	7.4. Summary of Important Points
	References
Chapter Eight: Object Measurement
	8.1. Introduction
	8.2. Measures for Binary Objects
		8.2.1. Size Measures
			8.2.1.1. Area
			8.2.1.2. Perimeter
			8.2.1.3. Area and Perimeter of a Polygon
		8.2.2. Pose Measures
			8.2.2.1. Centroid
			8.2.2.2. Orientation
		8.2.3. Shape Measures
			8.2.3.1. Thinness Ratio
			8.2.3.2. Rectangularity
			8.2.3.3. Circularity
			8.2.3.4. Euler Number
			8.2.3.5. Moments
				Central Moments
				Object Dispersion
				Rotationally Invariant Moments
				Zernike Moments
			8.2.3.6. Elongation
		8.2.4. Shape Descriptors
			8.2.4.1. The Differential Chain Code
			8.2.4.2. Fourier Descriptors
			8.2.4.3. The Medial Axis Transform
			8.2.4.4. Graph Representations
				Minimum Spanning Tree
				Delaunay Triangulation
	8.3. Distance Measures
		8.3.1. Euclidean Distance
		8.3.2. City-Block Distance
		8.3.3. Chessboard Distance
	8.4. Gray Level Object Measures
		8.4.1. Intensity Measures
			8.4.1.1. Integrated Optical Density
			8.4.1.2. Average Optical Intensity
			8.4.1.3. Contrast
		8.4.2. Histogram Measures
			8.4.2.1. Mean Gray Level
			8.4.2.2. Standard Deviation of Gray Levels
			8.4.2.3. Skew
			8.4.2.4. Entropy
			8.4.2.5. Energy
		8.4.3. Texture Measures
			8.4.3.1. Statistical Texture Measures
				The Gray Level Co-Occurrence Matrix
			8.4.3.2. Power Spectrum Features
	8.5. Object Measurement Considerations
	8.6. Summary of Important Points
	References
Chapter Nine: Object Classification
	9.1. Introduction
	9.2. The Classification Process
		9.2.1. Bayes Rule
	9.3. The Single-Feature, Two-Class Case
		9.3.1. A Priori Probabilities
		9.3.2. Conditional Probabilities
		9.3.3. Bayes Theorem
	9.4. The Three-Feature, Three-Class Case
		9.4.1. The Bayes Classifier
			9.4.1.1. Prior Probabilities
			9.4.1.2. Classifier Training
			9.4.1.3. The Mean Vector
			9.4.1.4. Covariance
			9.4.1.5. Variance and Standard Deviation
			9.4.1.6. Correlation
			9.4.1.7. The pdf
			9.4.1.8. Classification
			9.4.1.9. Log Likelihoods
			9.4.1.10. The Mahalanobis Distance Classifier
			9.4.1.11. Uncorrelated features
		9.4.2. A Numerical Example
	9.5. Classifier Performance
		9.5.1. The Confusion Matrix
	9.6. Bayes Risk
		9.6.1. The Minimum-Risk Classifier
	9.7. Relationships Among Bayes Classifiers
	9.8. The Choice of a Classifier
		9.8.1. Subclassing
		9.8.2. Feature Normalization
	9.9. Nonparametric Classifiers
		9.9.1. Nearest-Neighbor Classifiers
	9.10. Feature Selection
		9.10.1. Feature Reduction
			9.10.1.1. Principal Component Analysis
			9.10.1.2. Linear Discriminant Analysis
	9.11. Neural Networks
	9.12. Summary of Important Points
	References
Chapter Ten: Multispectral Fluorescence Imaging
	10.1. Introduction
	10.2. Basics of Fluorescence Imaging
		10.2.1. Image Formation in Fluorescence Imaging
	10.3. Optics in Fluorescence Imaging
	10.4. Limitations in Fluorescence Imaging
		10.4.1. Instrumentation-Based Aberrations
			10.4.1.1. Photon Shot Noise
			10.4.1.2. Dark Current
			10.4.1.3. Auxiliary Noise Sources
			10.4.1.4. Quantization Noise
			10.4.1.5. Other Noise Sources
		10.4.2. Sample-Based Aberrations
			10.4.2.1. Photobleaching
			10.4.2.2. Autofluorescence
			10.4.2.3. Absorption and Scattering of the Medium
		10.4.3. Sample and Instrumentation Handling-Based Aberrations
	10.5. Image Corrections in Fluorescence Microscopy
		10.5.1. Background Shading Correction
		10.5.2. Correction Using the Recorded Image
		10.5.3. Correction Using Calibration Images
			10.5.3.1. Two-Image Calibration
			10.5.3.2. Background Subtraction
		10.5.4. Correction Using Surface Fitting
		10.5.5. Autofluorescence Correction
		10.5.6. Spectral Overlap Correction
		10.5.7. Photobleaching Correction
	10.6. Quantifying Fluorescence
		10.6.1. Fluorescence Intensity and Fluorophore Concentration
	10.7. Fluorescence Imaging Techniques
		10.7.1. Immunofluorescence
		10.7.2. Fluorescence In Situ Hybridization (FISH)
		10.7.3. Quantitative Colocalization Analysis
		10.7.4. Fluorescence Ratio Imaging (RI)
		10.7.5. Fluorescence Resonance Energy Transfer (FRET)
		10.7.6. Fluorescence Lifetime Imaging (FLIM) FRET
			10.7.6.1. Time Correlated Single Photon Counting (TCSPC) FLIM-FRET
		10.7.7. Fluorescence Recovery After Photobleaching (FRAP)
		10.7.8. Total Internal Reflectance Fluorescence Microscopy (TIRFM)
		10.7.9. Fluorescence Correlation Spectroscopy (FCS)
	10.8. Summary of Important Points
	References
Chapter Eleven: Three-Dimensional Imaging
	11.1. Introduction
	11.2. Image Acquisition
		11.2.1. Wide-Field 3D Microscopy
		11.2.2. Confocal Microscopy
		11.2.3. Multiphoton Microscopy
		11.2.4. Microscope Configuration
		11.2.5. Other 3D Microscopy Techniques
	11.3. 3D Image Data
		11.3.1. 3D Image Representation
			11.3.1.1. 3D Image Notation
	11.4. Image Restoration and Deblurring
		11.4.1. The Point Spread Function
			11.4.1.1. Theoretical Model of the Psf
			11.4.1.2. Approximate Methods
		11.4.2. Models for Microscope Image Formation
			11.4.2.1. Poisson Noise
			11.4.2.2. Gaussian Noise
		11.4.3. Algorithms for Deblurring and Restoration
			11.4.3.1. No-Neighbor Methods
			11.4.3.2. Nearest-Neighbor Method
			11.4.3.3. Linear Methods
				Inverse Filtering
				Wiener Deconvolution
				Linear Least Squares
				Regularization
					Tikhonov Regularization
			11.4.3.4. Nonlinear Methods
				Jansson-van Cittert Method
				The Nonlinear Constrained Least Squares Method
				The Carrington Algorithm
				The Iterative Constrained Tikhonov-Miller Algorithm
			11.4.3.5. Maximum Likelihood Restoration
				The EM-ML Algorithm
				The Richardson-Lucy Algorithm
				Maximum Penalized Likelihood Method
				Maximum A Posteriori Method
			11.4.3.6. Blind Deconvolution
			11.4.3.7. Space-Variant Deconvolution
			11.4.3.8. Interpretation of Deconvolved Images
			11.4.3.9. Commercial and Free Deconvolution Packages
	11.5. Image Fusion
	11.6. Three-Dimensional Image Processing
	11.7. Geometric Transformations
	11.8. Pointwise Operations
	11.9. Histogram Operations
	11.10. Filtering
		11.10.1. Linear Filters
			11.10.1.1. Finite Impulse Response Filters
		11.10.2. Nonlinear Filters
			11.10.2.1. Median Filter
			11.10.2.2. Weighted Median Filter
			11.10.2.3. Minimum and Maximum Filters
			11.10.2.4. α-Trimmed Mean Filters
		11.10.3. Edge Detection Filters
	11.11. Morphological Operators
		11.11.1. Binary Morphology
		11.11.2. Grayscale Morphology
	11.12. Segmentation
		11.12.1. Point-Based Segmentation
		11.12.2. Edge-Based Segmentation
		11.12.3. Region-Based Segmentation
			11.12.3.1. Connectivity
			11.12.3.2. Region Growing
			11.12.3.3. Region Splitting And Merging
		11.12.4. Deformable Models
	11.13. Comparing 3D Images
	11.14. Registration
	11.15. Object Measurements in 3D
		11.15.1. Euler Number
		11.15.2. Bounding Box
		11.15.3. Center of Mass
		11.15.4. Surface Area Estimation
			11.15.4.1. Surface Estimation Using Superquadric Primitives
			11.15.4.2. Surface Estimation Using Spherical Harmonics
		11.15.5. Length Estimation
		11.15.6. Curvature Estimation
			11.15.6.1. The Surface Triangulation Method
			11.15.6.2. The Cross Patch Method
		11.15.7. Volume Estimation
		11.15.8. Texture
	11.16. 3D Image Display
		11.16.1. Montage
		11.16.2. Projected Images
			11.16.2.1. Voxel Projection
			11.16.2.2. Ray Casting
		11.16.3. Surface and Volume Rendering
			11.16.3.1. Surface Rendering
			11.16.3.2. Volume Rendering
		11.16.4. Stereo Pairs
		11.16.5. Color Anaglyphs
		11.16.6. Animations
	11.17. Summary of Important Points
	References
Chapter Twelve: Superresolution Image Processing
	12.1. Introduction
	12.2. The Diffraction Limit
	12.3. Deconvolution
		12.3.1. Signals and Noise
		12.3.2. Extrapolating Beyond the Diffraction Limit
			12.3.2.1. Statistical Methods
			12.3.2.2. Machine Learning Methods
	12.4. Superresolution Imaging Techniques
		12.4.1. Analytic Continuation
		12.4.2. Stimulated Emission Depletion Microscopy
		12.4.3. Expansion Microscopy
		12.4.4. Single Molecule Localization Microscopy
		12.4.5. Structured Illumination Microscopy
		12.4.6. Synthetic Superresolution with Machine Learning
	12.5. Summary of Important Points
	References
Chapter Thirteen: Localization Microscopy
	13.1. Introduction
		13.1.1. A Brief History of Localization Microscopy
	13.2. Overcoming the Diffraction Limit
		13.2.1. Diffraction-Limited Resolution
		13.2.2. Photoswitching Mechanisms
	13.3. Localizing Molecular Position
		13.3.1. Spot Candidate Selection
			13.3.1.1. Local Intensity Maxima
			13.3.1.2. Nonmaximum Suppression
			13.3.1.3. Centroid Estimation
			13.3.1.4. The Intensity Threshold
		13.3.2. Gaussian Model Fitting
			13.3.2.1. Least Squares Fitting
			13.3.2.2. The Method of Steepest Descent
			13.3.2.3. Newtons Method
			13.3.2.4. The Levenberg-Marquardt Method
			13.3.2.5. Maximum Likelihood Fitting
		13.3.3. Localization Methods
			13.3.3.1. Spot Centroid Calculation
			13.3.3.2. The Radial Symmetry Method
			13.3.3.3. Spline and Complex Model Fitting
		13.3.4. Visualization of Localization Data
			13.3.4.1. Scatterplots
			13.3.4.2. Two-dimensional Histograms
				Jittering
			13.3.4.3. Intensity Interpolation to Neighboring Pixels
				Averaged Shifted Histograms
			13.3.4.4. Gaussian Rendering
		13.3.5. Localization and Image Artifacts in SMLM
	13.4. Three-Dimensional Localization Microscopy
		13.4.1. Calibration Measurements
		13.4.2. Multiplane Imaging
		13.4.3. Point Spread Function Engineering
		13.4.4. Intensity-Based Approaches
			13.4.4.1. Supercritical Angle Localization
			13.4.4.2. Photometric Localization
	13.5. Quantitative Localization Microscopy
		13.5.1. Quality Control of Localization Data
			13.5.1.1. Temporal Drift Correction
				Fiducial Markers
				Self-Alignment
				Cross-Correlation Analysis
		13.5.2. Localization Precision and Image Resolution
			13.5.2.1. Theoretical Localization Precision
			13.5.2.2. Experimental Precision and Resolution
				Analyzing Isolated Emitter Spots
				Tracing and Tracking
				Localization Precision, Resolution, and Sampling
				Fourier-Ring Correlation
		13.5.3. Localization-Based Cluster Analysis
			13.5.3.1. Statistical SMLM Cluster Analysis
				Ripleys Functions
				Correlation-Based Clustering
			13.5.3.2. Density-Based Clustering (DBSCAN)
			13.5.3.3. K-means Clustering
			13.5.3.4. Voronoi Tessellation
			13.5.3.5. Bayesian Cluster Analysis
		13.5.4. Particle Averaging
	13.6. Implementation and Applications of SMLM
		13.6.1. Machine and Deep Learning for SMLM
		13.6.2. MINFLUX
		13.6.3. Applications of SMLM
	13.7. Summary of Important Points
	References
Chapter Fourteen: Motion Tracking and Analysis
	14.1. Introduction
	14.2. Image Acquisition
		14.2.1. Microscope Setup
		14.2.2. Spatial Dimensionality
		14.2.3. Temporal Resolution
	14.3. Image Preprocessing
		14.3.1. Image Denoising
		14.3.2. Image Deconvolution
		14.3.3. Image Registration
	14.4. Image Analysis
		14.4.1. Cell Tracking
			14.4.1.1. Cell Segmentation
			14.4.1.2. Cell Association
		14.4.2. Particle Tracking
			14.4.2.1. Particle Detection
			14.4.2.2. Particle Association
	14.5. Trajectory Analysis
		14.5.1. Geometry Measurements
		14.5.2. Diffusivity Measurements
		14.5.3. Velocity Measurements
	14.6. Sample Algorithms
		14.6.1. Cell Tracking
		14.6.2. Particle Tracking
	14.7. Summary of Important Points
	References
Chapter Fifteen: Deep Learning
	15.1. Introduction
		15.1.1. Basic Components of Neural Networks
		15.1.2. A Timeline of Convolutional Neural Network Development
		15.1.3. A Timeline of Deep Learning in Microscopy
	15.2. Deep Learning Concepts
		15.2.1. Training
		15.2.2. Activation Functions
		15.2.3. Cost Functions
		15.2.4. Convolutional Neural Networks
	15.3. Practical Applications
		15.3.1. Classification
		15.3.2. Detection
		15.3.3. Segmentation
	15.4. Software Frameworks
	15.5. Training Deep Learning Networks
		15.5.1. Data Augmentation
		15.5.2. Transfer Learning
	15.6. Application of Deep Learning for Cell Nuclei Detection
	15.7. Challenges
	15.8. Summary of Important Points
	References
Chapter Sixteen: Image Informatics
	16.1. Introduction
	16.2. Open-source Software Ecosystems
		16.2.1. Java Libraries and Tools
		16.2.2. Python Tools
		16.2.3. C++ Tools
		16.2.4. Tool Interoperation
	16.3. Image Acquisition
		16.3.1. Image Processing and Analysis
		16.3.2. Machine Learning Platforms
	16.4. Image Storage and Curation
		16.4.1. Data Curation
		16.4.2. Storage Backend
	16.5. Visualization
	16.6. Community
	16.7. Conclusion
	16.8. Summary of Important Points
	References
Glossary
	Further reading
Index
Back Cover