Benchmarking, Measuring, and Optimizing: Third BenchCouncil International Symposium, Bench 2020, Virtual Event, November 15–16, 2020, Revised Selected Papers (Lecture Notes in Computer Science)

Preface
Organization
BenchCouncil Rising Star Award Lecture
Scientific Benchmarking of Parallel Computing Systems
BenchCouncil Achievement Award Lecture
It’s a Random World: Learning from Mistakes, Errors, and Noise
Keynote
Benchmarking Quantum Computers
AIBench and Its Performance Rankings
DataBench Toolbox Supporting Big Data and AI Benchmarking
Contents
Best Paper Session
Characterizing the Sharing Behavior of Applications Using Software Transactional Memory
	1 Introduction
	2 Background
		2.1 Software Transactional Memory
		2.2 Sharing Behavior and Thread Mapping
		2.3 STAMP Applications
	3 Methodology of the Characterization
		3.1 Detecting Sharing in STM Applications
		3.2 Machine
		3.3 Mean Squared Error (MSE)
		3.4 Experiments
	4 Characterization of STM Sharing Behavior
		4.1 STM Memory Access Information
		4.2 Stability of Sharing Behavior Across Different Executions
		4.3 Stability of Sharing Behavior When Changing Input Parameters
		4.4 Stability of Sharing Behavior with Different Numbers of Threads
		4.5 Dynamic Behavior During Execution
		4.6 Overhead of the Behavior Detection
	5 Case Studies
		5.1 Performance Gains from Sharing-Aware Thread Mapping
		5.2 False Sharing in Kmeans
	6 Related Work
	7 Conclusion
	References
swRodinia: A Benchmark Suite for Exploiting Architecture Properties of Sunway Processor
	1 Introduction
	2 Background
		2.1 Architecture of Sunway Processor
		2.2 Parallel Programming Paradigms on Sunway Processor
		2.3 Rodinia Benchmark
	3 Porting and Optimizing Rodinia Benchmarks
		3.1 Back Propagation
		3.2 Breadth-First Search
		3.3 Hotspot3D
		3.4 Kmeans
		3.5 Needleman-Wunsch
		3.6 Pathfinder
	4 Evaluation
		4.1 Performance Comparison
		4.2 Roofline Analysis
	5 Insights
		5.1 Programmability vs. Performance
		5.2 Software Optimization
		5.3 Hardware Improvements
	6 Related Work
	7 Conclusion
	References
Data Management and Storage
Impact of Commodity Networks on Storage Disaggregation with NVMe-oF
	1 Introduction
	2 Background
		2.1 Storage Disaggregation and NVMe-oF
		2.2 Intel SPDK
	3 Characterization Methodology
	4 Evaluation
		4.1 Experimental Setup
		4.2 Bandwidth Evaluation
		4.3 Latency Evaluation
		4.4 CPU Utilization Study
		4.5 SSD Concurrency Study
	5 Discussion
	6 Related Work
	7 Conclusion and Future Work
	References
K2RDF: A Distributed RDF Data Management System on Kudu and Impala
	1 Introduction
	2 Related Works
		2.1 RDF
		2.2 SPARQL
		2.3 Related Systems
	3 Property Chain Model
		3.1 Model Definition
		3.2 Model Features
	4 System Design
		4.1 Data Loader
		4.2 Query Translater
		4.3 Query Executor
	5 Evaluation
		5.1 Environment
		5.2 Experimental Results
	6 Conclusion and Future Work
	References
Artemis: An Automatic Test Suite Generator for Large Scale OLAP Database
	1 Introduction
	2 Architecture Overview of Artemis
	3 Database Schema Generation
	4 Data Generation
		4.1 Deterministic Random Data Generation
		4.2 Parallel Data Generation
	5 Workload Generation
		5.1 Parameter Instantiation
	6 Oracle Generation
	7 Experiment
		7.1 Data Generation
		7.2 Query Generation
		7.3 Oracle Generation
	8 Related Work
	9 Conclusion
	References
OStoreBench: Benchmarking Distributed Object Storage Systems Using Real-World Application Scenarios
	1 Introduction
	2 Related Work
		2.1 Mainstream Distributed Object Storage System
		2.2 Benchmarks for Distributed Storage Systems
		2.3 Workloads Behavior and Storage Access Patterns
	3 Benchmark Building
		3.1 Requirements
		3.2 The Design of OStoreBench
		3.3 The Implementation of OStoreBench
	4 Evaluation
		4.1 Approach and Metrics
		4.2 Experimental Setup
		4.3 Experimental Results
	5 Conclusion
	6 Future Work
	References
ConfAdvisor: An Automatic Configuration Tuning Framework for NoSQL Database Benchmarking with a Black-box Approach
	1 Introduction
	2 Problem Statement
	3 System Design
		3.1 System Overview
		3.2 Bayesian Optimization
	4 Experiment Settings
		4.1 NoSQL Databases and Benchmarks
		4.2 Hardware Platforms
		4.3 Configuration Settings
	5 Experimental Results
		5.1 Impact of Operating System Parameters
		5.2 Performance Improvement
		5.3 Comparisons
	6 Related Work
	7 Conclusion
	References
Supercomputing
Optimization of the Himeno Benchmark for SX-Aurora TSUBASA
	1 Introduction
	2 Target Systems and Benchmark
		2.1 Overview of SX-Aurora TSUBASA
		2.2 Overview of the Himeno Benchmark
	3 Optimization of the Himeno Benchmark
		3.1 Optimizations for a Single VE
		3.2 Optimizations for Multiple VEs
	4 Evaluation
		4.1 Experimental Environment
		4.2 Evaluation of a Single VE
		4.3 Evaluation of Multiple VEs
		4.4 Power Efficiency
	5 Related Work
	6 Conclusions
	References
Benchmarking on GPU
Parallel Sorted Sparse Approximate Inverse Preconditioning Algorithm on GPU
	1 Introduction
	2 SPAI Algorithm
	3 Parallel SPAI on GPU
		3.1 Pre-GSPAI Stage
		3.2 Compute-GSPAI Stage
		3.3 Post-GSPAI Stage
	4 Evaluation and Analysis
		4.1 Effectiveness Analysis
		4.2 Performance Comparison
	5 Conclusion
	References
ComScribe: Identifying Intra-node GPU Communication
	1 Introduction
	2 Background
	3 Types of Communication
		3.1 Peer-to-Peer Communication
		3.2 Host-Device Communication
	4 Design and Implementation of ComScribe
		4.1 ComScribe Workflow
	5 Evaluation
		5.1 Micro-benchmarks from Comm|Scope and MGBench
		5.2 Overhead
		5.3 Macro-benchmarks from NVIDIA and MGBench
		5.4 Use-Cases: Deep Neural Network (DNN) Models
	6 Related Work
	7 Conclusion
	References
Application and Dataset
A Benchmark of Ocular Disease Intelligent Recognition: One Shot for Multi-disease Detection
	1 Introduction
	2 Related Work
		2.1 OCT Images Recognition
		2.2 Fundus Images Recognition
	3 Our Dataset
		3.1 Image Collection and Annotation
		3.2 Features of Dataset
		3.3 Split of Dataset
	4 Multi-disease Classification
		4.1 Network Structure
		4.2 Experimental Settings
		4.3 Experiment Analysis
	5 Discussion
	6 Conclusions
	References
MAS3K: An Open Dataset for Marine Animal Segmentation
	1 Introduction
	2 Related Work
		2.1 Underwater Image Enhancement
		2.2 Marine Animal Detection
		2.3 Marine Animal Camouflage
		2.4 Camouflaged Object Detection
	3 Proposed Dataset
		3.1 Image Collection
		3.2 Data Annotation
		3.3 Dataset Features and Statistics
	4 Benchmark Experiments
		4.1 Experimental Settings
		4.2 Results
	5 Conclusion
	References
Benchmarking Blockchain Interactions in Mobile Edge Cloud Software Systems
	1 Introduction
	2 Aspects in Benchmarking Blockchain Features for MECSS
		2.1 Mobile Edge Computing Software Systems
		2.2 Application-Required Features for Blockchain-Based MECSS
		2.3 Benchmarked Metrics
		2.4 MECSS Under Test and Infrastructures
		2.5 Coupling Software and Infrastructure Artefacts
	3 Framework for Benchmarks
		3.1 Overview
		3.2 Benchmark Functions
		3.3 Benchmark Experiment Configuration
		3.4 Prototype
	4 Experiments
		4.1 Flexibility in Benchmark Configurations
		4.2 Insights from Benchmarks for Developers
	5 Related Work
	6 Conclusions and Future Work
	References
Author Index