Influencing Factors in Speech Quality Assessment using Crowdsourcing

Abstract
Zusammenfassung
Acknowledgments
Contents
About the Author
Acronyms
1 Introduction
	1.1 Speech Quality
		1.1.1 Speech Quality Assessment
		1.1.2 Crowdsourcing
		1.1.3 Speech Quality Assessment in Crowdsourcing
		1.1.4 Differences Between Laboratory-Based and Crowdsourcing-Based Speech Quality Assessments
	1.2 Influencing Factors in Speech Quality Assessment Using Crowdsourcing
	1.3 Research Questions and Thesis Outline
2 Related Work
	2.1 Number of Stimuli
	2.2 Worker Performance and Task Repetition
	2.3 Environmental Background Noise
	2.4 Influence of Language Differences
	2.5 Conclusion
3 Method
	3.1 Laboratory Test
	3.2 Speech Database
		3.2.1 SwissQual 501
		3.2.2 SwissQual 502
		3.2.3 SwissQual P.501 Annex D
	3.3 Crowdsourcing Test
		3.3.1 Standardized Evaluation Method for Speech Quality in Crowdsourcing
		3.3.2 Crowdsourcing Platforms
		3.3.3 Test Setup and Procedure
			3.3.3.1 Qualification
			3.3.3.2 Training
			3.3.3.3 Assessment
		3.3.4 Environment
			3.3.4.1 Study Setup
			3.3.4.2 Audio Recording Setup
			3.3.4.3 Environment Video Recording
			3.3.4.4 Environment Questionnaire
			3.3.4.5 Audio Recording Analysis
			3.3.4.6 Questionnaire Results
			3.3.4.7 Discussion
	3.4 Simulated Crowdsourcing Test in Laboratory
	3.5 Result Metrics
	3.6 Conclusion
4 Test Structure
	4.1 Influence of Number of Stimuli
		4.1.1 Study Setup
			4.1.1.1 Speech Database
			4.1.1.2 Method
		4.1.2 Results
			4.1.2.1 Qualification
			4.1.2.2 Training and Assessment
			4.1.2.3 Influence of Number of Stimuli
		4.1.3 Discussion
	4.2 Impact of Task Repetition
		4.2.1 Study Setup
			4.2.1.1 Speech Material
			4.2.1.2 Laboratory Study
			4.2.1.3 Crowdsourcing Study
		4.2.2 Results
			4.2.2.1 Inter-Rater Reliability
			4.2.2.2 Intra-Rater Reliability
			4.2.2.3 Predicting Workers' Performance
		4.2.3 Discussion
	4.3 Conclusion
5 Impact of Background Noise
	5.1 Effect of Environmental Background Noise
		5.1.1 Study Setup
			5.1.1.1 Background Noise Signals
			5.1.1.2 Speech Database
		5.1.2 Results
			5.1.2.1 Laboratory vs. CSLvl0
			5.1.2.2 Influence of Background Noise
		5.1.3 Discussion
	5.2 Analysis of Noisy Speech Quality Scores Collected in Crowdsourcing Environments
		5.2.1 Speech Quality Scores
		5.2.2 Model
			5.2.2.1 Feature Selection
			5.2.2.2 Model Evaluation
			5.2.2.3 Model Tuning
		5.2.3 Discussion
	5.3 Environment Background Noise Classification
		5.3.1 Environment Background Noise Collection
			5.3.1.1 Dataset for Noise Classification
			5.3.1.2 Dataset for Noise Level Estimation
		5.3.2 Experiment BN1
			5.3.2.1 Results
		5.3.3 Experiment BN2
			5.3.3.1 Results
		5.3.4 Discussion
	5.4 Conclusion
6 Influence of Language
	6.1 Study Setup
		6.1.1 Speech Database
		6.1.2 Method
	6.2 Results
		6.2.1 Analysis of Laboratory vs. Studies E1, E2, and E3
		6.2.2 Influence of Language Differences
		6.2.3 Analysis of Conditions per Group
	6.3 Conclusion
7 Conclusion
Appendix A
	A.1 Speech Database SwissQual 501
Appendix B
	B.1 Speech Database SwissQual 502
Appendix C
	C.1 Speech Database SwissQual P.501 Annex D
References
Index