HPC, Big Data, and AI Convergence Towards Exascale: Challenge and Vision

Cover\nHalf Title\nTitle Page\nCopyright Page\nTable of Contents\nForeword\nForeword\nPreface\nPreface\nAcknowledgments\nEditors\nContributors\n1 Toward the Convergence of High-Performance Computing, Cloud, and Big Data Domains\n	1.1 Introduction\n		1.1.1 History of Cloud Computing\n		1.1.2 History of HPC\n		1.1.3 Evolution of Big Data\n		1.1.4 Evolution of Big Data Storage and Tools\n	1.2 Exploiting Convergence\n		1.2.1 CYBELE Project\n		1.2.2 DeepHealth Project\n		1.2.3 EVOLVE Project\n		1.2.4 LEXIS Project\n	Acknowledgment\n	References\n2 The LEXIS Platform for Distributed Workflow Execution and Data Management\n	2.1 Motivation\n	2.2 Architecture (Codesign) and Interfaces\n	2.3 Security\n	2.4 Accounting and Billing\n	2.5 Easy Access to HPC/Cloud Through a Specialized Web Portal\n	2.6 Market Analysis\n		2.6.1 LEXIS Project Impact\n	Acknowledgment\n	References\n3 Enabling the HPC and Artificial Intelligence Cross-Stack Convergence at the Exascale Level\n	3.1 Introduction\n	3.2 The Rise of Convergent Infrastructures\n	3.3 The ACROSS Approach to the HPC, Big Data, and AI Convergence\n		3.3.1 Heterogeneous Infrastructural Support\n		3.3.2 The Management of the Convergent Platform\n	3.4 Related Works\n	3.5 Conclusions\n	Acknowledgment\n	Notes\n	Bibliography\n4 Data System and Data Management in a Federation of HPC/Cloud Centers\n	4.1 Introduction: Data Federation of European HPC/Cloud Centers\n	4.2 Requirements On the LEXIS DDI\n		4.2.1 Unified Data Access\n		4.2.2 Usage and Federation of Diverse Data Backend Systems\n		4.2.3 Reliability and Redundancy\n		4.2.4 AAI Support\n		4.2.5 APIs\n		4.2.6 State-Of-The-Art Research Data Management\n	4.3 Federation Via a DDI Based On IRODS\n		4.3.1 Relevant Basic Properties of IRODS\n		4.3.2 IRODS HA Setup\n		4.3.3 IRODS Zones Federation Across Centers and Data Movement\n		4.3.4 Storage Tiering and Underlying Data Storage\n		4.3.5 Logical Structure of the DDI\n	4.4 Hardware\n		4.4.1 Storage Systems for HPC and Infrastructure-As-A-Service- Cloud Clusters\n		4.4.2 Storage Systems Dedicated to LEXIS\n		4.4.3 HPC–Cloud-Storage Interconnect and Data Node/Burst Buffer Concept\n			4.4.3.1 SBF (Smart Bunch of Flash)\n			4.4.3.2 SBB\n	4.5 Unified Access to the Platform Based On an AAI\n		4.5.1 LEXIS Identity and Access Management (IAM) Solution, SSO, and AAI\n		4.5.2 Platform Services Vs. AAI: Separation of Concerns\n		4.5.3 LEXIS DDI and IAM/AAI System\n	4.6 Data Management Via APIs\n		4.6.1 Data Search, Upload, and Download APIs\n		4.6.2 Staging API\n		4.6.3 Replication and PID Assignment API\n		4.6.4 Helper APIs\n		4.6.5 Compression/Decompression/Encryption/Decryption API\n	4.7 Integration With EUDAT Services\n		4.7.1 EUDAT B2HANDLE\n		4.7.2 EUDAT B2SAFE\n		4.7.3 EUDAT B2STAGE\n	4.8 Conclusion\n	Acknowledgment\n	References\n5 Distributed HPC Resources Orchestration for Supporting Large-Scale Workflow Execution\n	5.1 Introduction\n	5.2 Federated Execution Platforms\n	5.3 WMSs and Implementation in LEXIS\n		5.3.1 Dynamic Workflow Orchestration\n		5.3.2 Resource Management Metrics\n	5.4 Workflow Data Management\n	5.5 LEXIS Pilot Use Cases and Orchestration\n	5.6 Related Works\n	5.7 Conclusion\n	Acknowledgment\n	Notes\n	Bibliography\n6 Advanced Engineering Platform Supporting CFD Simulations of Aeronautical Engine Critical Parts\n	6.1 Introduction: Background and LEXIS Aeronautics Pilot\n	6.2 Engineering Case Studies in the LEXIS Aeronautics Pilot\n	6.3 The Turbomachinery Case Study\n		6.3.1 Engineering Context\n		6.3.2 Digital Technology Deployment\n			6.3.2.1 Application Workflow\n			6.3.2.2 Main Application Software and HW Resources\n		6.3.3 First Results\n		6.3.4 Benefit–Cost Analysis of HW Acceleration\n		6.3.5 Next Steps\n	6.4 The Rotating Parts Case Study\n		6.4.1 Engineering Context\n		6.4.2 Digital Technology Deployment\n			6.4.2.1 Application Workflow\n			6.4.2.2 Main Application Software and HW Resources\n		6.4.3 First Results\n			6.4.3.1 SPH Liquid-Phase Simulation\n			6.4.3.2 SPH Gas-Phase Simulation\n		6.4.4 Next Steps\n	6.5 Final Remarks\n	Acknowledgment\n	Notes\n	References\n7 Event-Driven, Time-Constrained Workflows: An Earthquake and Tsunami Pilot\n	7.1 Introduction\n	7.2 Event-Driven, Time-Constrained Workflows\n		7.2.1 Requirements\n		7.2.2 Background\n		7.2.3 Overall View of the Workflow\n	7.3 Workflow Components\n		7.3.1 Shakemap and Exposure Dataset\n		7.3.2 Tsunami Simulations\n		7.3.3 SEM\n	7.4 Technological Layers\n		7.4.1 Technology Layer 1: Orchestration\n		7.4.2 Technology Layer 2: Heterogeneous Compute\n		7.4.3 Technology Layer 3: Data\n	7.5 Conclusion\n	Note\n	References\n8 Exploitation of Multiple Model Layers Within LEXIS Weather and Climate Pilot: An HPC-Based Approach\n	8.1 Introduction: Background and Driving Forces\n	8.2 The Weather and Climate Pilot\n	8.3 Observational Data\n	8.4 LEXIS DDI and Weather and Climate Data API\n	8.5 LEXIS Orchestration System\n	8.6 Weather and Climate Pilot Workflows\n		8.6.1 WRF–ERDS Workflow Examples\n	8.7 Conclusion\n	Acknowledgment\n	References\n9 Data Convergence for High-Performance Cloud\n	9.1 Introduction\n	9.2 Motivations\n	9.3 Design and Implementation\n	9.4 Karvdash\n	9.5 DataShim\n		9.5.1 Overview\n		9.5.2 Dataset Custom Resource Definition\n		9.5.3 DatasetInternal Custom Resource Definition\n		9.5.4 DataShim Operator and Admission Controller\n		9.5.5 Caching Plugin\n		9.5.6 Objects Caching On CEPH\n		9.5.7 Ceph-Based Caching Plugin Implementation\n		9.5.8 Evaluation of the Ceph-Based Caching Plugin\n	9.6 H3\n		9.6.1 Overview\n		9.6.2 Data and Metadata Organization\n		9.6.3 The H3 Ecosystem\n	9.7 Integration\n	9.8 Related Work\n	9.9 Conclusions\n	Note\n	References\n10 The DeepHealth HPC Infrastructure: Leveraging Heterogenous HPC and Cloud-Computing Infrastructures for IA-Based Medical Solutions\n	10.1 Introduction\n	10.2 The Parallel Execution of EDDL Operations\n		10.2.1 COMPSs\n		10.2.2 StreamFlow\n	10.3 Cloud Infrastructures\n		10.3.1 Hybrid Cloud\n		10.3.2 Parallel Execution On Cloud Environments\n			10.3.2.1 Parallel Cloud Execution Based On COMPSs\n			10.3.2.2 Parallel Cloud Execution Based On StreamFlow\n	10.4 Acceleration Devices: GPU and FPGAs\n		10.4.1 FPGA Acceleration\n			10.4.1.1 The DeepHealth FPGA Infrastructure\n			10.4.1.2 An Optimized FPGA Board Design for DL\n			10.4.1.3 FPGA-Based Algorithms\n		10.4.2 Many Core and GPU Acceleration\n	10.5 Conclusions\n	Notes\n11 Applications of AI and HPC in the Health Domain\n	11.1 Introduction\n	11.2 AI and HPC in the Health Domain in 2020\n	11.3 DeepHealth Concept\n	11.4 DeepHealth Use Cases\n	11.5 Use of HPC and Cloud in Medical Pilots\n		11.5.1 UC2 – UNITOPatho\n		11.5.2 UC3 – UNITOBrain\n		11.5.3 UC4 – Chest\n		11.5.4 UC5 – UNITO Deep Image Annotation\n		11.5.5 UC12 – Skin Cancer Melanoma Detection\n	11.6 DeepHealth Value Proposition\n	11.7 Conclusions\n	Notes\n12 CYBELE: On the Convergence of HPC, Big Data Services, and AI Technologies\n	12.1 Introduction: Background and Driving Forces\n	12.2 Identified Gaps: Motivating the CYBELE Vision\n	12.3 Materializing the Solution: Convergence of HPC, Big Data, and AI\n		12.3.1 Data and Infrastructure Access Security Layer\n		12.3.2 Embedded Experiments Composition Layer\n		12.3.3 Parallel and Distributed Execution Management Layer\n		12.3.4 Data Services Layer\n		12.3.5 Visualization and Reporting Layer\n	12.4 Key Takeaways and Conclusions\n	Note\n	References\n13 CYBELE: A Hybrid Architecture of HPC and Big Data for AI Applications in Agriculture\n	13.1 Introduction: Vision and Challenges\n	13.2 Background\n		13.2.1 AI in Big Data Analytics On Cloud\n		13.2.2 AI On HPC Systems\n	13.3 Hybrid Big Data and HPC Resource for AI Applications in CYBELE\n	13.4 Parallelization and Deployment of AI Applications On HPC Systems\n		13.4.1 Pilot Soybean Farming\n			13.4.1.1 Pilot Description\n			13.4.1.2 Application Parallelization for HPC Systems\n		13.4.2 Pilot Wheat Ear\n			13.4.2.1 Pilot Description\n			13.4.2.2 Application Parallelization for HPC Systems\n	13.5 Performance Evaluation for Pilot Soybean Farming and Pilot Wheat Ear\n	13.6 Discussion\n	13.7 Conclusion Remarks and Future Works\n	Acknowledgments\n	Notes\n	References\n14 European Processor Initiative: Europe’s Approach to Exascale Computing\n	14.1 Introduction\n	14.2 European Processor Initiative\n		14.2.1 Global Technical Panstream\n		14.2.2 GPP Stream\n		14.2.3 Accelerator Stream\n		14.2.4 Automotive Stream\n	14.3 Conclusion\n	Acknowledgment\n	This Work Has Received Funding From the European Union’s Horizon 2020 Research and Innovation Programme “European Processor Initiative (EPI)” Under Grant Agreement No 826647.\n	Bibliography\nIndex