Computerized Adaptive Testing: From Concept to Implementation (Methodology in the Social Sciences Series)

Cover
Half Title Page
Series Page
Title Page
Copyright
Dedication
Series Editor’s Note
Preface
Contents
Part I. The Why, What, and How of CAT
	1. Why CAT?
		What Is CAT?
		Benefits of CAT
			Psychometric Benefits
			Benefits for Examinees
			Organizational Benefits
		Challenges of CAT
			Organizational Challenges
			Examinee Challenges
			Psychometric Challenges
		When Is CAT Feasible and When Is It Not?
	2. The Birth of Adaptive Testing and Its Current Status
		Binet’s Adaptive Test
		The Evolution of Adaptive Testing into CAT
		Early Approaches to CAT
			Two-Stage and Multistage Tests
			CATs from “Pyramidal” Item Structures
			The Self-Scoring Flexilevel Test
			The Stratified-Adaptive Test
			Deficiencies of the Early Approaches to CAT
		The Current Status of Applications of CAT
	3. Item Response Theory Used in CAT
		Item Response Functions for Dichotomously Scored Items
		Option Response Functions for Polytomously Scored Items
		Information Functions
			Item Information Functions
			Test and Bank Information Functions
		Measuring People with IRT
			Maximum Likelihood Estimation of O
			Variations of Maximum Likelihood Estimation
			Linking and Equating
		Estimating IRT Item Parameters
			Selecting a Software Package
			Interpreting the Output
		Summary
		Advanced Topic: Item Parameter Estimation for Complex Models
	4. Using CAT to Measure Individual Differences
		The Components of a Measurement CAT
			The Item Bank
			Beginning a CAT
			Selecting the Next Item
			Scoring Examinees
			Ending the CAT
		Putting It All Together: How CAT Works
			Sample CAT: Examinee A
			Sample CAT: Examinee B
		Some Properties of Fully Adaptive IRT-Based CATs
		Advanced Issues
			CAT Item Selection
			Ending a CAT
	5. CAT for Classification and Classification Plus Measurement
		Setting the Cutscore
			The Bank Response Function
		Types of Classification Applications
			Testing for Classification Only
			Adaptive Classification Test (ACT)
			Testing for Classification Plus Measurement
		Advanced Issues
			Cutscore- versus Estimate-Based Item Selection
			Ending an SCT When a O Estimate Is Too Close to the Cutscore
			Classifying with More Than Two Cutscores
			Classifying from Multiple Measurements
Part II. Getting Your CAT Operational
	6. Feasibility and Planning
		Developing a Business Case
		The Item Bank
			Options and Costs
		Costs of Test Delivery
			Delivering High-Stakes CATs
			Custom Test Center Costs
		Potential Risks
			Psychometric Risks
			Nonpsychometric Risks
		Additional Considerations
			Developing a Project Plan
			Stakeholder Buy-In
			Implementing Simulations to Facilitate CAT Design
		Conclusions
	7. Developing a CAT Item Bank: Writing and Evaluating Items
		How Many Items Do I Need in My Bank?
			Write as Many Items as You Can
			Do Not Build an Item Bank Solely for Today—Think about the Future as Well
		What Kinds of Items Work for CAT?
		Developing High-Quality Items and Item Banks
			Planning
			Item Writing
			Evaluating Your Items
			Maintaining Your Item Bank
		Characteristics of a Good Item Bank
		Further Information
	8. Pretesting and Linking Your Items, and Adding New Items to an Existing Bank
		Pretesting Your Items
		IRT Item (and Person) Parameter Linking
			Conversion Methods of Linking
			Full-Metric Item Parameter Linking
			Seeding New Items into a CAT Item Bank
		Additional Considerations in Item Bank Development
		Advanced Topic: Automated Item Generation
	9. Using Simulations to Design a CAT
		Post-Hoc or Real-Data Simulation
			Making a Conventional Test More Efficient
		Designing a CAT from a CAT Item Bank
			Monte Carlo Simulations
		Hybrid Simulations
			How Much Missing Data?
		Conclusions
	10. Implementing Your CAT
		CAT Delivery Software
			Free CAT Delivery Software
			Commercial CAT Delivery Software
			Build-Your-Own CAT Delivery System
		How Can I Get CAT Operational for My Organization?
			Stakeholder Buy-In
			Other Resources
Part III. Applications, Special Types of CATs, Special Issues, and a Potential Future for CAT
	11. Examples of Operational CAT Programs
		CAT in Education
			K–12 Formative/Interim/Summative Assessment
		CAT for Military Accessioning
			Armed Services Vocational Aptitude Battery (CAT-ASVAB)
			Navy Computer Adaptive Personality Scales (NCAPS)
			Higher Education Admissions and Placement
			Computerized Adaptive Screening Test (CAST)
		CAT for Pre-Employment Testing
			Adjustable Competence Evaluation (ACE)
			Computerized Adaptive Test for Ability (CATA)
		CAT for Certification and Licensure
			Certified Public Accountant (CPA) Exam
			Emergency Medical Services (EMS) Exams
			North American Pharmacist Licensure Examination (NAPLEX)
			Multistate Pharmacy Jurisprudence Examination (MPJE)
			National Council Licensure Examinations (NCLEX)
		CAT in Healthcare
			Mental Health
			Patient-Reported Outcomes (PROs)
		Additional Applications of CAT
	12. Other Types of CATs
		Multistage Tests
			Two-Stage Tests
			MSTs
			Summary
		CAT for Cognitive Diagnosis
			Cognitive Diagnostic Assessment
			Cognitive Diagnostic CAT (CD-CAT)
			Summary
		CAT for Measuring Multiple Variables
			Multidimensional IRT
			Multidimensional CAT (MD-CAT)
			Summary
	13. Special Issues
		Constrained CATs
			Content Balancing
			Item Exposure
			Multiple Constraints: Shadow Tests
		CAT Using Items with Multiple Questions per Stimulus
		CAT Administration Issues
			Fixed-Length CATs
			Time Limits
		The Examinee Experience
			A Suggested Preamble to a CAT
	14. Adaptive Measurement of Change: Evaluating the Psychometric Significance of Measured Individual Change
		Identifying Significant Individual Change Using an Early Adaptive Measurement of Change (AMC) Method
			The AMC Procedure
			The Likelihood Ratio Index (LRI)
		Example AMC Applications
			Univariate Change
			Multivariate (Profile) Change
		AMC Research
			Discussion and Conclusions
		Conclusions
	15. Some Pathways toward Future CATs and CAT Research
		Item Banking
			AIG
			On-the-Fly CAT
			Chatbots as Item Writers
		Test Delivery
			Examinee Responses
			More Realistic Test Items
		Supplementing and Improving Scores
			Response Times
			Game-Based Tests
			Other Process Data
		Integration of Testing and Instruction
		A Path for Future CAT Research: Does CAT Improve Validity?
		A Sample Application of CAT SEMs in Predicting External Criteria
			Method
			Results
			Summary and Conclusions
		Leveraging Current and Future Technology to Improve Testing
Software Appendices
	A. Software Packages for Estimating IRT Parameters
	B. Software Packages for CAT Simulations
	C. Software Platforms for CAT Delivery
Acronyms
References
Author Index
Subject Index
About the Authors
Symbols