Performance Evaluation and Benchmarking: 14th TPC Technology Conference, TPCTC 2022, Sydney, NSW, Australia, September 5, 2022, Revised Selected Papers

Preface
TPCTC 2022 Organization
Contents
Pick & Mix Isolation Levels: Mixed Serialization Graph Testing
	1 Introduction
	2 Serialization Graph Testing
		2.1 Protocol Description
		2.2 Algorithmic Adjustments
		2.3 Many-Core Optimizations
	3 Mixing in the Wild
	4 Mixing Theory
		4.1 System Model
		4.2 Weak Isolation Levels
		4.3 Mixing of Isolation Levels
	5 Mixed Serialization Graph Testing
		5.1 Protocol Design
		5.2 Discussion
		5.3 Implementation Details
	6 Evaluation
		6.1 Isolation
		6.2 Contention
		6.3 Update Rate
		6.4 Scalability
	7 Conclusion
	References
BoDS: A Benchmark on Data Sortedness
	1 Introduction
	2 Data Sortedness Metrics
	3 Generating (K,L)-Sorted Data
	4 The Benchmark on Data Sortedness
	5 BoDS in Action
		5.1 Raw Ingestion Performance
		5.2 Mixed Workload Performance
	6 Toward Sortedness Awareness
	7 Conclusion
	References
Disaggregated Database Management Systems
	1 Introduction
	2 Hardware Disaggregation
		2.1 Fungible's DPU-Based Disaggregation
		2.2 Liqid's Composable Disaggregated Infrastructure (CDI)
	3 Memory Disaggregation
	4 Disaggregated Database Management Systems
		4.1 AlloyDB
		4.2 Rockset
		4.3 Nova-LSM
	5 Future Research Directions
	References
TPCx-AI on NVIDIA Jetsons
	1 Introduction
	2 Background
		2.1 Resource-Aware Machine Learning
		2.2 System-on-Chip Devices
		2.3 TPCx-AI Benchmark
	3 Related Work
	4 Experimental Methodology and Setup
		4.1 Systems
		4.2 Metrics
		4.3 Benchmark Suite Modifications
	5 Results
		5.1 Whole Benchmark Run
		5.2 Time-Breakdown per Use Case
		5.3 Use Case 8
	6 Discussion
		6.1 Machine Learning on Jetsons
		6.2 TPCx-AI Benchmark for Edge Devices
	7 Conclusion
	References
More the Merrier: Comparative Evaluation of TPCx-AI and MLPerf Benchmarks for AI
	1 Introduction
	2 AI Benchmarking Tools
	3 MLPerf AI Benchmarks
		3.1 MLPerf Training
		3.2 MLPerf Inference
	4 TPCx-AI Benchmark
		4.1 TPCx-AI Metric
		4.2 TPCx-AI Metric Analysis
	5 MLPerf vs TPCx-AI
		5.1 Scope and Scoring
		5.2 Results Review
		5.3 Code License
		5.4 Cost
		5.5 Efficiency Scores
		5.6 Accelerators
		5.7 When to Use Each Tool
	6 Summary and Conclusions
	References
Preliminary Scaling Characterization of TPCx-AI
	1 Introduction
	2 Related Work
	3 TPCx-AI Kit
		3.1 Licensing and Setup
		3.2 Configuration
		3.3 Benchmark Execution
	4 Performance Results
		4.1 System Under Test (SUT)
		4.2 Single-node Implementation
		4.3 Multinode Implementation
	5 Conclusions and Future Work
	References
4mbench: Performance Benchmark of Manufacturing Business Database
	1 Introduction
	2 Description of the Manufacturing Business Based on the 4m Model
	3 4mbench
		3.1 Database Schema
		3.2 Business Case and Dataset Generation
		3.3 Test Queries
	4 Experimental Study
		4.1 Dataset Generation and Loading
		4.2 Test Queries
		4.3 4mQ.3 on Different Settings
	5 Related Work
	6 Conclusion
	A Experiment Configuration
		A.1 SQL Description of Test Query (4mQ.3)
		A.2 [DATE] Variable Substitution
		A.3 Configuration Parameters of PostgreSQL
	References
Benchmarking Considerations for Trustworthy and Responsible AI (Panel)
	1 Introduction
	2 Current State of AI Benchmarking
	3 Deconstructing Trust and Responsibility in AI
	4 Metrics for Trust and Responsibility
	5 Challenges and Opportunities for Benchmarking Trust and Interpretability
	6 Summary and Conclusions
	References
TPCx-AI: First Adopter’s Experience Report
	1 TTA
		1.1 Background of TTA
		1.2 Test Description
	References
New Initiatives in the TPC
	1 Introduction
		1.1 Venturing into New Benchmark Domains
		1.2 TPC's Benchmark Development Model Until 2013
		1.3 Express BenchmarksTM, a Benchmark Model for Rapid Benchmark Development
	2 TPC Express BenchmarksTM Becoming Reality
	3 Big Data Benchmarks
		3.1 Virtualization Benchmarks, TPCx-V and TPCx-HCI
		3.2 TPCx-IoT
		3.3 Enterprise and Express Class Publications
	4 TPC Derived Benchmarks
		4.1 Initiative to Allow the Use of TPC Benchmark Material in Non-TPC Benchmarks
		4.2 TPC's Open Source Initiative
	5 TPCx-AITM, First End-To-End AI benchmark Standard
		5.1 Allowing Cloud Based Benchmark Publications
	6 Conclusion
	References
Author Index