Big Data in Astronomy: Scientific Data Processing for Advanced Radio Telescopes

Front matter
Copyright
Contributors
Preface
Acknowledgments
Introduction to radio astronomy
	The history of astronomy
		Ancient astronomy
		Astronomy from the mid-16th century to the mid-19th century
		Astronomy since the mid-19th century
	What is radio astronomy
		How does radio astronomy occur
		The radio stars, quasars, and black holes
			The strongest radio source, Cygnus A, in the sky
			The discovery of cliff allergens and radio galaxies
			Nonthermal radiation
			Synchronous radiation
			Synchrotron radiation pattern
			Connect nonthermal radiation and cosmic rays
			Astrophysics of cosmic rays
			Discovery of quasars
		The radio astronomy instrument: Radio telescope
		Some achievements of radio astronomy
		Astronomical research nowadays
	Advanced radio telescope
		The square kilometer array (SKA)
		Fast
	The challenge of radio astronomy
		System noise
		Antennas and collecting area
		Data transmission
	The development tendency of radio astronomy
		Mid-frequency aperture arrays
		Entering a near future
	References
Fundamentals of big data in radio astronomy
	Big data and astronomy
		Background of big data
		Definitions and features of big data
		Development of big data
		Big data in astronomy
		Statistical challenges in astronomy
	Increasing data volumes of telescopes
		Sloan digital sky survey
		Visible and infrared survey telescope for astronomy
		Large synoptic survey telescope
		Thirty meter telescope
	Existing methods for the value chain of big data
		Data generation
		Data acquisition
		Data storage
		Data analysis
			Traditional data analysis methods
			Big data analytic methods
			Architecture for big data analysis
	Current statistical methods for astronomical data analysis
		Nonparametric statistics
		Data smoothing
		Multivariate clustering and classification
		Nondetections and truncation
		Spatial point processes
	Platforms for big data processing
		Horizontal scaling platforms
		Vertical scaling platforms
			High performance computing (HPC) clusters
			Multicore CPU
			Graphics processing unit (GPU)
			Field programmable gate arrays (FPGA)
	References
Preprocessing pipeline on FPGA
	FPGA interface for ADC
		ADC interleaving
		Bit alignment
		Stream deserialization
	FIR filtering
		Leakage
		Scalloping loss
		Polyphase filter
	Time-frequency domain transposing
		Real-valued FFT
		Demultiplexing
	Correlators based on FPGA
		FPGA-based correlator for SMA radio telescope
		FPGA-based correlator of ALMA radio telescope
	General architectures for data reduction design and implementation
		Software design analysis
		Hardware implementation hierarchy
		A case study of a typical compression design
			Workflow of data reduction based on FPGA
			Data communication interface
			Design issues in pipeline
	Conclusion
	References
Real-time stream processing in radio astronomy
	Introduction
	Stream processing
	Heterogeneous signal processing
		Common architectures
	Ethernet interconnect
		High-speed Ethernet technologies
		TCP/IP
		TCP and UDP
		UDP datagram structure
		Multicast
	First-stage data processing
		Data rates
		Channelization
	Data redistribution
		The corner-turn problem
		Data backplanes
		Packetized Ethernet interconnect
	Second-stage processing
		Performance modeling
		Packet capture
		Kernel bypass packet capture
		Ring buffers
		CPU/GPU pipeline frameworks
		Disk I/O
		Performance tuning
	Discussion
		Future outlook
	Acknowledgments
	References
Digitization, channelization, and packeting
	Digitization
	Channelization
	Packeting
	References
Processing data of correlation on GPU
	Introduction
	GPU-based cross-correlator engines
		General processing steps of software implementation
		Software architecture of GPU-based implementation
	Applying and implementing gridding algorithm after cross-correlator
		Gridding algorithm application in SKA
		Gridding algorithm analysis
		Parallel implementation of gridding/degridding algorithms and analysis of experimental results after cross-correlator
			Time overhead analysis
			Parallelization implementation of gridding/degridding algorithm
			Performance of gridding/degridding on GPU after cross-correlator
	Applying and implementing deconvolution algorithm and parallel implementation after cross-correlator
		CLEAN algorithm
		Parallel implementation of CLEAN algorithm
		Performance of CLEAN algorithm on GPU
	Summary
	References
Flux calibration for single-dish radio telescopes
	Basic concepts
		Antenna temperature
		System noise temperature
		Telescope gain and effective area
		System equivalent flux density
	Flux calibration
		Calibrating with SEFD
		Calibrating with noise diode
		Gain curve
	Processing spectral line data
		Example of SIGGMA data
		Example of FAST data
	Observations of a brown dwarf by Arecibo single dish
		Basic information and previous observations of TVLM 513
		Instrument, observation, and data reduction
		Results
			May 2008
			June 2009
	References
Imaging algorithm optimization for scale-out processing
	Imaging process
		Make a dirty image
		The relationship between the dirty image and the sky intensity distribution
	Gridding and degridding
		Gridding process
		Aliasing
		Degridding
	The choice of the gridding function in the era of big data
		Spheroidal function and least-misfit gridding function
		Tabulating the gridding function for image processing
		Gridding computational cost in the big data era
	Bayesian source discrimination
		An application: Bayesian source discrimination
		Applications to big data radio interferometry
	References
Execution framework technology
	Introduction
	OpenCluster
		Fundamental model
		OpenCluster implementation
			Factory
			Workshop
			Worker
			Manager
			WorkPeice
		Deployment and operation
		Stability
		Hybrid resource scheduling
	DALiuGE
		DALiuGE works mode
		Develop
		Compose
		Select and parameterize
		Translate
		Deploy
			Resource mapping
			Drop managers
			Physical graph deployment
		Execute
		Drop
			Drop channels
			Drop I/O
	Acknowledgments
	References
Application design for execution framework
	OpenCluster applications design
	MUSER pipeline using OpenCluster
		Data file format transformation
		Real-time imaging and monitoring
	Design CHILES on AWS using DALiuGE
		Setup
		Results and costs
	The migration of SAGECal/MPI to DALiuGe
		About SAGECal
		Code analysis
		Workload characterization
		Drop wrapping
		Dynamic MPI process control
		Process location
		Dataflow and graphs
			Coarse-grained graphs
			Fine-grained graphs
			Implementation with BashShellApp
			Implementation with DynlibApp
	Acknowledgments
	References
	Further reading
Heterogeneous computing platform for backend computing tasks
	Introduction
	Computing architecture and platform
		Graphical processing unit
			GALARIO
			Precision
		Many integrated core
		Cell broadband engine
		ASIC/FPGA
	Algorithm benchmarking
		Gridding
		Deconvolution
			Clean
			Richardson-Lucy method
			MEM
			NNLS
			Sparse regularization
		Compressed sensing
		Source extraction
	Telescopes and applications
		SKA
		SETI
		MUSER
		Parkes observatory
		Radio transient detection
		Solar flare detection
	Conclusion
	References
	Further reading
High-performance computing for astronomical big data
	Introduction
	Execution framework and prototype test
		High performance execution frameworks
			Parallel programming models
			Distributed computing frameworks
			Dataflow computation model
		DALiuGE scalabilty test on supercomputers
			DALiuGE prototyping test by ASTRON
				Hardware
				Environment setup
				Pipeline graph execution
				Execution results
			DALiuGE scalability test on Tianhe-2 supercomputer
				Hardware
				Environment setup
				Pipeline development and execution
				DALiuGE execution results
	Improving SKA algorithm reference library on high-performance computing platform
		ARL computational kernel
		Simple imaging pipeline
		Functional imaging algorithms
	Summary
	References
Spark and dask performance analysis based on ARL image library
	Introduction
	Preliminaries and notations
		Spark
		Dask
		Genetic algorithm
		Pipeline use case
	Experiment
		End-to-end performance
		Individual step performance
	Task scheduling based on data processing capacity
		GA on task scheduling
		Computation topology model on task scheduling
	Network connection model and routing topology model
	Conclusion
	References
Applications of artificial intelligence in astronomical big data
	Introduction
	Machine learning for astronomical data calibration and repair
		Clustering analysis algorithm for missing values: KSC
		PCA-based machine learning for classification of SDSS transient survey images
		CCD defect inspection with artificial neural network
	Artificial intelligence algorithms in astronomy data classification and preprocessing
		Morphological classification of galaxies
			Supervised learning method
			Unsupervised learning
		Star/galaxy classification and detection
			Supervised learning
			Unsupervised learning
		Spectral analysis
			Artificial neural network
			Deep learning
		Cosmic ray classification
			Forward neural networks
			Clustering algorithm
	Artificial intelligence application in astronomy data analysis
		Photometric redshift
			Multilayer perceptron and artificial neural network
			Bayes algorithm
			Convolutional neural network
		Flare detection
			Artificial neural network
			Support vector machine
			Deep learning
		Galaxy parameter analysis
			Machine learning algorithms
			Deep learning algorithms
		Periodicity analysis
			Artificial neural network
			Clustering algorithm
		Supervised learning for detection of dispersed radio pulses
		Unsupervised learning for estimating extinction
	Conclusion
	References
Mapping the universe with 21cm observations
	The neutral hydrogen and 21cm line
		The physics of spin temperature
		The evolution of 21cm signal through cosmic history
	The 21cm experiments
		HI galaxy survey
		Intensity map observations
		The 21cm tomography experiments
		The 21cm global spectrum experiments
	Data processing
		Imaging and beam forming
		Foreground
		The foreground wedge
	Conclusion
	References
	Further reading
Index
	A
	B
	C
	D
	E
	F
	G
	H
	I
	J
	K
	L
	M
	N
	O
	P
	Q
	R
	S
	T
	U
	V
	W
	X
	Y
	Z