Virtual Standard Setting: Setting Cut Scores

Cover
Table of contents
List of figures
List of tables
List of acronyms
Chapter 1: Introduction
	1.1 Overview of the study
	1.2 Scope of the study
	1.3 Outline of the chapters
Chapter 2: Literature review
	2.1 Background to standard setting
	2.2 The importance of setting valid cut scores
		2.2.1 Standard setting methods
			2.2.1.1 Examples of test-centred methods
				Variants of the Angoff method
				The Bookmark method
				The Objective Standard Setting (OSS) method
			2.2.1.2 Examples of examinee-centred methods
				The Borderline Group (BG) method and the Contrasting Group (CG) method
				The Body of Work (BoW) method
		2.2.2 Evaluating and validating standard setting methods
	2.3 Standard setting in language assessment
		2.3.1 Current LTA standard setting research
			2.3.1.1 The first publicly available CEFR alignment studies
			2.3.1.2 Studies investigating understanding of method or CEFR
			2.3.1.3 Studies investigating external validity evidence
			2.3.1.4 Studies proposing new methods/modifications
	2.4 Challenges associated with standard setting
		2.4.1 Theoretical and practical challenges
		2.4.2 Logistics
	2.5 Virtual standard setting
		2.5.1 Virtual standard setting: Empirical studies
		2.5.2 Challenges associated with virtual standard setting
	2.6 Media naturalness theory
		2.6.1 Re-evaluating virtual standard setting studies through MNT
	2.7 Summary
Chapter 3: Methodology
	3.1 Research aim and questions
	3.2 Methods
		3.2.1 Embedded MMR design
		3.2.2 Counterbalanced workshop design
		3.2.3 Instruments
			3.2.3.1 Web-conferencing platform and data collection platform
			3.2.3.2 Test instrument
			3.2.3.3 CEFR familiarisation verification activities
			3.2.3.4 Recruiting participants
			3.2.3.5 Workshop surveys
			3.2.3.6 Focus group interviews
			3.2.3.7 Ethical considerations
	3.3 Standard setting methodology
		3.3.1 Rationale for the Yes/No Angoff method
		3.3.2 Pre-workshop platform training
		3.3.3 In preparation for the virtual workshop
		3.3.4 Description of the workshop stages
			3.3.4.1 Introduction stage
			3.3.4.2 Orientation stage
				3.3.4.2.1 CEFR familiarisation verification activity A
				3.3.4.2.2 CEFR familiarisation verification activity B
				3.3.4.2.3 Familiarisation with the test instrument
			3.3.4.3 Method training stage
			3.3.4.4 Judgement stage
				Round 1 Stage
				Round 2 Stage
				Round 3 Stage
	3.4 Data analysis methods and frameworks
		3.4.1 CEFR verification activities analysis
		3.4.2 Internal validity of cut scores
			Classical test theory (CTT)
			Rasch measurement theory (RMT)
			The many-facet Rasch measurement (MFRM) model
		3.4.3 Comparability of virtual cut score measures
		3.4.4 Differential severity
		3.4.5 Survey analysis
		3.4.6 Focus group interview analysis
	3.6 Summary
Chapter 4: Cut score data analysis
	4.1 Cut score internal validation: MFRM analysis
		4.1.1 Rasch group level indices
		4.1.2 Judge level indices
	4.2 Cut score internal validation: CTT analysis
		4.2.1 Consistency within the method
		4.2.2 Intraparticipant consistency
		4.2.3 Interparticipant consistency
		4.2.4 Decision consistency and accuracy
			The Livingston and Lewis method
			The Standard Error method
	4.3 Comparability of cut scores between media and environments
		4.3.1 Comparability of virtual cut score measures
		4.3.2 Comparability of virtual and F2F cut score measures
	4.4 Differential severity between medium, judges, and panels
		4.4.1 Differential judge functioning (DJF)
		4.4.2 Differential medium functioning (DMF)
		4.4.3 Differential group functioning (DGF)
	4.5 Summary
Chapter 5: Survey data analysis
	5.1 Survey instruments
	5.2 Perception survey instrument
		5.2.1 Evaluating the perception survey instruments
		5.2.2 Analysis of perception survey items
			Qualitative comments for communication item 1
			Audio medium
			Video medium
			Qualitative comments for communication item 2
			Audio medium
			Video medium
			Qualitative comments for communication item 3
			Audio medium
			Video medium
			Qualitative comments for communication item 4
			Qualitative comments for communication item 5
			Audio
			Video medium
			Qualitative comments for communication item 6
			Audio medium
			Video medium
			Qualitative comments for communication item 7
			Audio medium
			Video medium
			Qualitative comments for communication item 8
			Audio medium
			Video medium
			Qualitative comments for communication item 9
			Audio medium
			The video medium
	5.3 Procedural survey items
		5.3.1 Evaluating the procedural survey instruments
	5.4 Summary
Chapter 6: Focus group interview data analysis
	6.1 Analysis of transcripts
	6.2 Findings
		6.2.1 Psychological aspects
			Distraction in the video medium
			Self-consciousness in the video medium
			Lack of non-verbal feedback in the audio medium
			Inability to distinguish speaker in the audio medium
			Inability to discern who was paying attention in audio medium
			Cognitive strain in the audio medium
		6.2.2 Interaction
			Lack of small talk in virtual environments
			No digression from the topic in virtual environments
			Differences in amounts of discussion between virtual and F2F settings
		6.2.3 Technical aspects
			Technical problems in virtual environments
			Turn-taking system
		6.2.4 Convenience
			Time saved in virtual environments
			Freedom to multi-task in virtual environments
			Less fatigue in virtual environments
		6.2.5 Decision-making in virtual environments
	6.3 Summary
Chapter 7: Integration and discussion of findings
	7.1 Research questions
		7.1.1 Research questions 1, 2, and 3
		7.1.2 Research question 4
		7.1.3 Research question 5
	7.2 Limitations
	7.3 Summary
Chapter 8: Implications, future research, and conclusion
	8.1 Significance and contribution to the field
	8.2 Guidance for conducting synchronous virtual cut score studies
		Demands for facilitators and/or co-facilitators
		Establishing a virtual standard setting netiquette
		Selecting a suitable virtual platform
		Selecting an appropriate medium for the workshop
		Recruiting online participants
		Training in the virtual platform
		Uploading materials
		Monitoring progress and engaging judges
	8.3 Recommendations for future research
	8.4 Concluding remarks
Appendices
Appendix A CEFR verification activity A (Key)
Appendix B Electronic consent form
Appendix C Judge background questionnaire
Appendix D Focus group protocol
	Introductory statement
	Focus group interview questions
Appendix E Facilitator’s virtual standard setting protocol
Appendix F CEFR familiarisation verification activity results
Appendix G: Facets specification file
Appendix H: Intraparticipant consistency indices
Appendix I: Group 5 group level and individual level Rasch indices
Appendix J: Form A & Form B score tables
Appendix K: DJF pairwise interactions
Appendix L: DGF pairwise interactions
Appendix M: Wright maps
References
Author index
Subject index
Series Index