Benchmarking, Measuring, and Optimizing. 14th BenchCouncil International Symposium, Bench 2022 Virtual Event, November 7–9, 2022 Revised Selected Papers

Preface
Organization
Invited Talks
	BenchCouncil Achievement Award Lecture: Benchmarking: An Incomparable Science?
	BenchCouncil Rising Star Award Lecture: Rethinking Benchmarking in AI: Evaluation-as-a-Service and Dynamic Adversarial Data Collection
	Towards Fair Machine Learning with Imperfect Information
	Contents
Architecture and System
A Quantitative Analysis of OpenMP Task Runtime Systems
	1 Introduction
	2 Related Work
	3 OMPTB: The OpenMP Task Benchmark
	4 Experimental Setup
	5 Experimental Results
		5.1 General Experimental Factors
		5.2 Homogeneous Workloads
		5.3 Homogeneous Case: Correlation Analysis
		5.4 Heterogeneous Workloads
	6 Conclusions
	References
EAIBench: An Energy Efficiency Benchmark for AI Training
	1 Introduction
	2 Related Work
		2.1 Benchmark
		2.2 Metrics
	3 Methodology and Implementation
		3.1 Metric and Measurement
		3.2 Benchmarks and Configuration Space
		3.3 Monitor and Analyzer
		3.4 System Under Test
	4 Experiment and Result
		4.1 Experimental Methodology
		4.2 Stability of Epochs
		4.3 Model Parameters Analysis
		4.4 GPU Comparison
		4.5 Different Layers Analysis
	5 Conclusion
	References
MSDBench: Understanding the Performance Impact of Isolation Domains on Microservice-Based IoT Deployments
	1 Introduction
	2 Related Work
	3 Benchmark Design
	4 Empirical Evaluation
		4.1 Microbenchmark Results
		4.2 End to End Benchmark Results
		4.3 Throughput Results
	5 Conclusion
	References
Algorithm and Dataset
ShoeMaster: A Benchmark for Sketch2Image Translation of Shoes
	1 Introduction
	2 Related Work
		2.1 Fashion Datasets
		2.2 Sketch2Image Translation
	3 The ShoeMaster Benchmark
		3.1 Shoes Category Knowledge Hierarchy
		3.2 Dataset Construction
		3.3 Evaluation Metric
	4 Experiments
		4.1 Approaches
		4.2 Methodology
		4.3 Results and Analysis
	5 Conclusion
	References
Open Source Software Supply Chain Recommendation Based on Heterogeneous Information Network
	1 Related Work
		1.1 GitHub Open Source Data Collection and Mining
		1.2 GitHub Project Recommendation
		1.3 Recommendation Methods Based on Heterogeneous Information Networks
	2 Proposed Framework
		2.1 Data Preprocessing
		2.2 Oswhin Construction
		2.3 Metapath Selection
		2.4 Oswhin-Based Embedding
		2.5 Personalized Recommendation
	3 Experiment
		3.1 Dataset
		3.2 Evaluation Metrics
		3.3 Experimental Results and Analysis
	4 Conclusion
	References
BasicTS: An Open Source Fair Multivariate Time Series Prediction Benchmark
	1 Introduction
	2 Related Works
	3 Benchmark Building
		3.1 Design Thoughts
		3.2 Implementation of BasicTS
	4 Evaluation
		4.1 Experimental Setup
		4.2 Experimental Results
		4.3 Review
	5 Conclusion
	6 Future Works
	References
Benchmarking Object Detection Models with Mummy Nuts Datasets
	1 Introduction
	2 Background and Motivation
	3 Requirements of Detecting Mummy Nuts with CNNs
		3.1 Overview of Object Detection Models
		3.2 Requirements of Benchmarking Models
	4 Benchmarking Methodology
		4.1 Proposed Datasets: Original and Augmented
		4.2 Metrics
		4.3 Proposed Tools
	5 Experiments
		5.1 Platform Selection
		5.2 Results
		5.3 Observations and Discussion
	6 Related Work
	7 Conclusion and Future Work
	References
Network and Memory
An Analysis of Long-Tailed Network Latency Distribution and Background Traffic on Dragonfly+
	1 Introduction
		1.1 Motivations
		1.2 Contributions
	2 Related Work
	3 Experimental Setup
		3.1 Computing
		3.2 Network
		3.3 Microbenchmarks and Applications
	4 Network Latency Distribution Analysis
		4.1 Job Placement Locality and Performance Variability
	5 Background Traffic Analysis
		5.1 Background Traffic Heuristic
		5.2 Correlation Analysis
		5.3 The Impact of Background Traffic on Long-Tail
		5.4 Application Analysis
	6 Discussion
	7 Conclusion
	References
MCCBench: A C10M Benchmark Oriented to Interactive Network Services
	1 Introduction
	2 Motivation
		2.1 Benchmark Design
		2.2 Benchmark Implementation
	3 MCCBench Design
		3.1 Workload Definition
		3.2 Service Framework
		3.3 Performance Evaluation Criteria
	4 MCCBench Implementation
		4.1 Workloads
		4.2 Architecture
		4.3 Case Study
	5 Evaluation
		5.1 Experiment Setup
		5.2 Results
		5.3 Expansion Experiment
	6 Conclusion
	References
STAMP-Rust: Language and Performance Comparison to C on Transactional Benchmarks
	1 Introduction
	2 Background and Related Work
		2.1 The STAMP Benchmark Suite
		2.2 Software Transactional Memory in Rust
	3 STAMP on Safe Rust
		3.1 Type-Level Safety
		3.2 Composable Transactions
		3.3 The Overhead of Safety
		3.4 The Complexity of Using Associative Arrays
	4 Analysis of Automatically Generated Benchmarks
	5 Evaluation
		5.1 Methodology
		5.2 Performance Comparison
		5.3 Qualitative Analysis
	6 Conclusion
	References
Author Index