Local Language Testing: Practice across Contexts (Educational Linguistics, 61)

Contents
Contributors
Local Language Testing Practice across Contexts
	1 Test Purpose(s)
	2 Local Language Tests as Policy Instruments
	3 Assessment of Language for Specific Purposes
	4 Languages, Contexts, Scope
	5 Language Assessment Literacy
	6 Insights Gained
	References
Using Multi-faceted Rasch Analysis to Examine Raters, Prompts, and Rubrics in an Office-Hour Role-Play Task
	1 Introduction
		1.1 The Oral English Test (OET)
	2 Testing Problem Encountered
	3 Literature Review
	4 Methods
		4.1 Participants and Role-Play Prompts
		4.2 Data Analysis
	5 Results and Discussion
		5.1 The Wright Map
		5.2 Rubric Criteria
		5.3 Prompts
		5.4 Prompt and Rater Interaction
	6 Insights Gained
		6.1 OET Raters
		6.2 The Prompts
		6.3 The Rubric
	7 Conclusion
	Appendices
		Appendix A: Role-Play Prompts
			Task Four: Office Hours Role-Play
				Role-Play Scenarios
					# 1: (Attendance Policy)
					# 2: (Chronic Lateness)
					# 3: (Extension Request)
					# 4: (Makeup Test Request)
		Appendix B: OET Rubric
	References
Validation of the Czech Language Certificate Exam with Respect to the Local Context
	1 Introduction
	2 Test Purpose and Testing Context
	3 Testing Problem Encountered
	4 Literature Review
	5 Methods, Approaches, and Solutions
		5.1 Analysis of the CEFR and Construct Revision
		5.2 Standard Setting
		5.3 Familiarization
		5.4 Standard Setting Method for Receptive Skills: Direct Consensus Method
			Validity Checks
		5.5 Standard Setting Method for Productive Skills: Body of Work Method
			Data Collection
			Panel Composition
			Design
			Validity Checks
	6 Assembling Comparable Test Forms
	7 Insights Gained
	References
Speaking “CEFR” about Local Tests: What Mapping a Placement Test to the CEFR Can and Can’t Do
	1 Introduction
	2 Local Testing Context
		2.1 The University Context
		2.2 Local Test Purpose
		2.3 The Evaluation Scales
		2.4 ESL Program Structure and the CEFR
		2.5 Other University Stakeholders
	3 Testing Problem Encountered
	4 Literature Review
	5 Methods
		5.1 Linkages Between the ANG and the CEFR
		5.2 Mapping Procedure
			Familiarisation Sessions
			Specification
			Standardization Training
			Standard Setting
			The Tucker-Angoff Method – The Minimally Acceptable Person (MAP)
	6 Results and Discussion
		6.1 Productive Skills – Writing
		6.2 Productive Skills – Speaking
		6.3 Comprehension Skills – Reading
		6.4 Comprehension Skills – Listening
	7 Insights Gained
		7.1 Benefits for Test Revision and Quality Control
		7.2 Benefits for Stakeholders
	8 Implications for Test Developers and Users
	References
Designing a New Diagnostic Reading Assessment for a Local Post-admission Assessment Program: A Needs-Driven Approach
	1 Introduction: Test Purpose and Testing Context
	2 Testing Problems Encountered
	3 Literature Review
		3.1 Diagnostic Language Assessments
		3.2 Principles for Defining the Constructs for a Diagnostic Language Assessment
		3.3 Academic Reading
	4 Methods
	5 Results and Discussion
		5.1 Construct Definition
		5.2 Refining the Construct Model
		5.3 Operationalization
	6 Insights Gained
	7 Implications for Test Development and Use
	References
Pre-post Elicited Imitation: Documenting Student Gains in the Purdue Language and Cultural Exchange
	1 Introduction: Context, Purpose, and Problem Encountered
	2 Testing Problem Encountered
	3 Literature Review
		3.1 Elicited Imitation: What It Measures
		3.2 Pre-Post Score Gains: As Validity Evidence for Test and as Instructional Sensitivity Metric for Items
	4 Methods
		4.1 Instrument: The ACE-In EI Module
		4.2 Participants
		4.3 Data Collection
		4.4 Data Analysis
			CTT Analyses of Measurement Qualities
			Pre-post Score Comparisons
			Item Analysis: Instructional Sensitivity
	5 Results and Discussion
		5.1 Interrater Reliability in the EI Pre- and Post-test Section-Total Rater-Scores
		5.2 Test Form Comparability
		5.3 Item Analysis in a CTT Framework: Item Difficulty and Discrimination
		5.4 Item Analysis in a CTT Framework: Internal Consistency and Standard Error of Measurement
		5.5 ACE-in EI Pre- and Post-test Score Comparisons
		5.6 A Close-Up View of Individual EI Items’ Instructional Sensitivity
	6 Implications for Test Developers and Users
	Appendices
		Appendix A. Elicited Imitation (“Listen and Repeat” in ACE-In) Instructions and Warm-Up Item
		Appendix B. Elicited Imitation (a.k.a. “Listen and Repeat” in ACE-In) Rating Scale
	References
Exploring the Alignment of Test Tasks with Curriculum Goals among Finnish 6th Graders
	1 Introduction
		1.1 Research Questions
	2 Alignment
	3 Assessing Young Learners’ Language Skills
	4 The National Core Curriculum as a General Guideline
	5 Methods
	6 Results and Discussion
	7 Insights Gained
	Appendices
		Appendix A: Objectives of Instruction in the A-Syllabus in English in Grades 3–6 (NAE, p. 241)
		Appendix B: The Evolving Language Proficiency Scale for Level A2.1
	References
When Student Background Overrides Score-Based Placement: Tension Between Stakeholders’ Conceptualizations of Fairness
	1 Test Purpose and Testing Context
		1.1 Placement Test Procedures
		1.2 How the Content Subjects Work
	2 Testing Problem Encountered
	3 Literature Review
	4 Methods
		4.1 Placement and Subject Marks Data
		4.2 Student Views
			Survey of Placement Experience
			Focus Groups
		4.3 Academic Staff Focus Groups
	5 Results and Discussion
		5.1 RQ1 Enrolment Numbers
		5.2 RQ2 Final Subject Marks
		5.3 RQ3. Perceived Accuracy of Placement
	6 Insights Gained
	7 Implications for Test Developers and Score Users
	References
Local Tests, Local Contexts: The Italian Language Testing Scenario
	1 Introduction: Test Purpose and Testing Context
		1.1 Integration: Basic Principles and Action Strategies
		1.2 The Role of Language in Integration Procedures
		1.3 Ministerial Guidelines for Test Development
	2 Problem Encountered
	3 Methods
	4 Results and Discussion
	5 Insights Gained
	6 Implications for Test Developers and Users
	References
Identifying the Phonological Errors of Second-Modality, Second-Language (M2-L2) Novice Signers Through Video-Based Mock Tests
	1 Introduction: Test Purpose and Testing Context
	2 Testing Problem Encountered
	3 Literature Review
		3.1 Phonological Parameters of BSL
		3.2 Causes of Phonological Errors
		3.3 Concept of Mock Tests
	4 Methods
	5 Results and Discussion
	6 Insights Gained
	7 Conclusions: Implications for Test Developers and Users
	References
An Indigenous Rubric for Assessing Dispensing Drugs in English of Thai Pharmacy Students: Reliability and Perception Issues
	1 Introduction
	2 Test Purpose and Test Context
	3 Testing Problem Encountered
	4 Literature Review
		4.1 Indigenous Criteria in English for Specific Purposes
		4.2 Tests for Language for Specific Purposes
	5 Methods
		5.1 Research Questions
		5.2 Participants
		5.3 Indigenous Rubric Development
		5.4 Criteria Selection
		5.5 Weighted Scores
		5.6 Data Collection and Analysis
	6 Results and Discussion
		6.1 RQ1 Can Language Instructors Reliably Use the Indigenous Rubric to Rate Student’s Dispensing Performance?
		6.2 RQ2 What Are the Language Teachers’ Perceptions of the Rubric?
		6.3 Problems and Suggestions
	7 Insights
	Appendix
		Questions for Raters
		Rubric
	References
Using Spoken Dialog Systems to Assess L2 Learners’ Oral Skills in a Local Language Testing Context
	1 Introduction: Test Purpose and Testing Context
	2 Testing Problem Encountered
	3 Literature Review
		3.1 Oral Communication of ESP Learners in the Hospitality Context
		3.2 Role-Play as a Test Task in ESP Assessment
		3.3 SDS Tasks for Oral Proficiency Assessment
	4 Methods
		4.1 Participants
		4.2 Materials and Instruments
		4.3 Rating Scale and Scoring
		4.4 Test Administration
			Surveys and Semi-structured Interviews
			Data Analysis
	5 Results and Discussion
		5.1 Test Takers’ Perceptions of the CTEOC Tasks
			Overall Opinions About the Tasks
			Potential Test Impact
			Task Authenticity
			Overall Preference of a Partner
			Self-Evaluation of Performance
			Suggestions for Potential Task Improvements
			Raters’ Perceptions of the Rating Scale
	6 Insights Gained
	7 Implications for Test Developers and Users
	Appendices
		Appendix A
			Sample Transcript of a Test Task
		Appendix B
			Rating Scale
		Appendix C
			Test Takers’ Questionnaire Responses in ‘Overall’ and ‘Preference’ Categories
	References