Evaluating Human Service Outcomes

Preface
Contents
About the Author
Chapter 1: The Essence of Outcome Evaluation
	Introduction
	Objectives
	Types of Human Service Evaluation
		Outcome Evaluation
		Other Types of Human Service Evaluation
			The Evaluation of Human Need
			The Evaluation of Service Quantity
			The Evaluation of Service Quality
			The Evaluation of Service Efficiency
	The Four Main Purposes of Human Service Research
		The Evaluation of Services
		The Description of People
		The Explanation of Things
		The Exploration of the Unknown
	The Process of Outcome Evaluation
		Step 1: Determine the Research Question and Study Purpose
		Step 2: Develop a Knowledge Base for the Study
		Step 3: Design the Evaluative Study
		Step 4: Collect and Analyze Data
		Step 5: Draw Conclusions
		Step 6: Describe the Service that Was Evaluated
		Don’t Put the Cart Before the Horse!
	Evidence-Based Practice as a Guide
	This Book
		The Organization of This Book
	Summary
Chapter 2: Developing Your Knowledge Base
	Introduction
	Objectives
	Steps in the Process of Developing Your Knowledge Base
		Step 1: Presenting the Scope of Your Review
		Step 2: Finding Your Sources
		Step 3: Reviewing Your Sources
			Levels of Evidence
		Step 4: Writing Your Literature Review
	Summary
	References
Chapter 3: Developing the Methods for Your Outcome Study
	Introduction
	Objectives
	Selecting Your Study Sample and Generalizing Your Study Findings
		Types of Samples
		Sampling Error
		Two Ways to Generalize Your Study Results
	Measuring Your Study Variables
		Defining Your Study Variables
		Qualitative and Quantitative Forms of Measurement
		Reliability and Validity in the Measurement of Psychosocial Variables
		Finding a Published Scale
		Designing Your Own Measurement Scale
			Defining the Variables You Are Measuring
			Constructing the Items for the Measurement Device
	Determining Your Research Design
		Causes of the Clients’ Measured Growth
		Group Research Designs
			One Group Pretest-Posttest Design
			Comparison Group Design
		Single-Subject Research Designs
		Single-Subject Research Designs that Fail to Control for Maturation
			The Limited AB Single-Subject Design
			The B Single-Subject Design
		Single-Subject Research Designs that Control for Maturation
			The AB Single-Subject Research Design
	Composing Your Study Hypothesis
	Summary
	References
Chapter 4: Collecting and Analyzing Your Data
	Introduction
	Objectives
	Collecting Data
		Collecting Data from Human Subjects in an Ethical Manner
		Recording Your Data
		Developing Your Data Plan
	Selecting the Statistic for Your Outcome Study
		Preliminary Steps for Testing Your Study Hypothesis
		The Six Data Situations for Outcome Research
		Data Situations that Do Not Fit
	Selecting a Statistic for Describing Clients
		Common Descriptive Statistics for Data Recorded Numerically
		Common Descriptive Statistics for Categorical Data
	Selecting a Statistic for Explaining Client Gain
	Analyzing Your Data
	Reporting Your Results
	Summary
Chapter 5: Using the Internet to Analyze Your Outcome Data
	Introduction
	Objectives
	Preliminary Steps
		Organizing Your Data
		The Six Data Situations for Outcome Evaluative Research
	Comparing Matched Pretest and Posttest Scores
		Example
	Steps in the Process of Comparing Matched Scores
	Comparing a Set of Scores to a Single Score
		Example
	Steps in the Process of Comparing a Set of Scores to a Single Score
	Comparing the Gain Scores of Two Groups
		Example
	Steps in the Process of Comparing the Scores of Two Groups
	Comparing Two Groups on the Basis of a Dichotomous Variable
		Example
	Comparing Multiple Treatment Scores to a Single Baseline Score for One Client
		Example
	Steps in the Comparison of Multiple Scores to a Single Score
	Comparing Multiple Treatment Scores to Multiple Baseline Scores for a Single Client
		Example
	Steps in the Comparison of Multiple Treatment Scores to Multiple Baseline Scores
	Summary
	Practice Assignment
		Research Case Examples
Chapter 6: Using SPSS to Analyze Your Outcome Data
	Introduction
	Objectives
	Preliminary Steps
		Organizing Your Data
		The Six Data Situations for Evaluative Studies
	Comparing Matched Pretest and Posttest Scores
		Example
	Steps in the Comparison of Matched Scores
		Step 1: Establishing Your Data File
		Step 2: Entering Your Data
		Step 3: Analyzing Your Data
		Step 4: Reporting Your Results
	Comparing a Set of Scores to a Single Score for a Group of Clients
		Example
	Steps in the Comparison of a Set of Scores to a Single Score
		Step 1: Establishing Your Data File
		Step 2: Entering Your Data
		Step 3: Analyzing Your Data
		Step 4: Reporting Your Results
	Comparing the Gain Scores of Two Groups
		Example
	Steps in the Comparison of the Scores of Two Groups
		Step 1: Establishing Your Data File
		Step 2: Entering Your Data
		Step 3: Analyzing Your Data
		Step 4: Reporting Your Results
	Comparing Two Groups on the Basis of a Dichotomous Variable
		Example
	Steps in the Comparison of Two Groups Using a Dichotomous Variable
		Step 1: Establishing Your Data File
		Step 2: Entering Your Data
		Step 3: Analyzing Your Data
		Step 4: Writing Your Results
	Comparing Multiple Treatment Scores to a Single Baseline Score for One Client
		Demonstration Example
	Steps in the Comparison of a Set of Scores to a Single Score for One Client
		Step 1: Establishing Your Data File
		Step 2: Entering Your Data
		Step 3: Analyzing Your Data
		Step 4: Reporting Your Results
	Comparing Multiple Treatment Scores to Multiple Baseline Scores for a Single Client
	Steps in the Comparison of Multiple Treatment Scores to Multiple Baseline Scores for a Single Client
		Step 1: Establishing Your Data File
		Step 3: Analyzing Your Data
		Step 4: Reporting Your Results
	Summary
	Practice Assignment on the Testing of the Evaluative Hypothesis
Chapter 7: Using Special Excel Files to Analyze Your Outcome Data
	Introduction
	Objectives
	Preliminary Steps
		Organizing Your Data
		The Six Data Situation for Evaluative Studies
	Comparing Matched Pretest and Posttest Scores
		Example
	Steps in Comparing Matched Scores
		Step 1: Organizing Your Data
		Step 2: Entering Your Data
		Step 3: Analyzing Your Data
		Step 4: Reporting Your Results
	Comparing a Set of Scores to a Single Score for a Group of Clients
		Example
	Steps in Comparing a Set of Scores to a Single Score
		Step 1: Composing Your Data
		Step 2: Entering Your Data
		Step 3: Analyzing Your Data
		Step 4: Reporting Your Results
	Comparing the Gain Scores of Two Groups
		Demonstration Example
	Steps in Comparing a Set of Scores for Two Groups
		Step 1: Composing Your Data
		Step 2: Entering Your Data
		Step 3: Analyzing the Data
		Step 4: Reporting Your Results
	Comparing Two Groups on the Basis of a Dichotomous Variable
		Demonstration Example
	Steps in Comparing Two Groups on the Basis of a Dichotomous Variable
		Step 1: Recording Your Data
		Step 2: Analyzing the Results
		Step 3: Reporting Your Results
	Comparing Multiple Treatment Scores to a Single Baseline Score for One Client
		Demonstration Example
		Step 1: Entering the Data into the Excel File
		Step 2: Analyzing the Data
		Step 3: Reporting the Results
	Comparing Multiple Treatment Scores to Multiple Baseline Scores for a Single Client
		Example
	Steps in the Comparison of Multiple Treatment Scores to Multiple Baseline Scores
		Analyzing the Data
		Step 3: Reporting Your Results
	Summary
	Practice Assignment on the Testing of the Evaluative Hypothesis
Chapter 8: Describing Clients
	Introduction
	Objectives
	The Process of Descriptive Research
		Step 1: Determine the Purpose of the Study
		Step 2: Select the Study Sample
		Step 3: Decide What to Describe
		Step 4: Collect Your Data
		Step 5: Select the Statistic for Each Variable
	Analyzing Descriptive Data
	Using the Internet to Compute Descriptive Statistics
		Step 1: Enter the Calculator Soup Webpage
		Step 2: Enter Your Data into the Blank Box on the Screen
		Step 3: Calculate the Descriptive Statistics
	Reporting the Results of Your Descriptive Study
	Summary
	References
Chapter 9: Explaining Client Outcomes
	Introduction
	Objectives
	The Issue of Causation
	The Steps in the Explanatory Research Process
		Step 1: Determine the Purpose, Research Question, and Knowledge Base
		Step 2: Develop Your Explanatory Research Hypothesis
		Step 3: Collect and Record Your Data
		Step 4: Analyze Your Data
		Step 5: Report Your Results
	Analyzing Your Explanatory Data
		Data Situations Previously Addressed in This Book
		Additional Data Situations for Explanatory Research
		Data Situation A: Using the Correlation Coefficient to Examine the Relationship Between Two Interval Variables
		Example
			Option 1: Using the Internet as the Mechanism for Computing the Correlation Coefficient
			Option 2: Using SPSS to Compute the Correlation Coefficient
		Reporting the Results and Conclusions for the Correlation Example
		Data Situation B: Using ANOVA When You Are Comparing the Scores of Several Groups
		Example
			Option 1: Using the Internet to Test Your Hypothesis Using ANOVA
			Option 2: Using SPSS to Test Your Hypothesis Using ANOVA
		Reporting the Results of the Study Using ANOVA
	Summary
	Reference
Chapter 10: Getting Ideas on How to Improve Service Through Qualitative Surveys
	Introduction
	Objectives
	Quantitative and Qualitative Measurement
		The Essence of Qualitative Research
			Exploratory Research and Qualitative Measurement
		The Nature of Qualitative Data
		Approaches to Qualitative Research
	The Social Survey
	Steps in the Process of Conducting a Social Survey
		Step 1: Determine the Purpose of the Survey
		Step 2: Select Your Study Sample
		Step 3: Design Your Measurement Tool
		Step 4: Administer the Survey
		Step 5: Analyze Data
		Step 6: Draw Conclusions
	One Model for Content Analysis of Qualitative Data
		Step 1: First-Level Coding
		Step 2: Credibility Assessment of First-Level Codes
		Step 3: Second-Level Coding
		Step 4: Enumeration of Second-Level Codes
		Other Steps
		Drawing Conclusions
	An Exercise in the Content Analysis of Qualitative Data
		Description of the Study
		The 2019 Cohort
		The 2022 Cohort
		Tasks in the Content Analysis of These Data
	Summary
Chapter 11: Writing Your Research Report
	Introduction
	Objectives
	Reporting the Purpose of Your Study and the Knowledge Base
		Reporting Your Study Purpose and Research Question
		Reporting Your Knowledge Base
	Reporting Your Study Methods
		Reporting Your Study Sample
		Describing Your Measurement Tools
		Stating Your Study Hypothesis
		Describing Your Research Design
		Writing Your Results and Conclusions
	Describing the Service Being Evaluated
		Describing the Objectives of the Service
		Describing the Structure of the Service
		Describing the Personnel of the Service
		Describing the Model of the Service
		Summary of Your Description of the Service
	Summary
	References
Chapter 12: Facing the Challenges for Outcome Evaluation
	Introduction
	How Can We Justify the Resources Expended for Our Services?
	Is Our Knowledge Base a Sufficient Guide?
	Can We Generalize Our Findings on a Logical Basis?
	Does Our Measurement Tool Pass the Test of Face Validity?
	Do We Know that Our Service Was Delivered According to Promise?
	Does Our Research Design Have to Control for Normal Growth over Time?
	Why Should We Be Concerned with Statistical Significance?
	How Do We Know if We Have Practical Significance?
	Are Our Conclusions Consistent with Our Data?
	Did We Put the Cart Before the Horse?
	Summary
	References
Appendix: Inventory of Critical Research Concepts (N = 111)
	AB Single-Subject Research Design
	Alternative Treatment Design
	Analysis of Variance (ANOVA)
	B Single-Subject Research Design
	Bar Chart
	Baseline Period
	Causation
	Chance
	Cherry Picking
	Chi Square
	Code in Content Analysis of Qualitative Data
	Comparison Group
	Comparison Group Research Design
	Content Analysis in Qualitative Research
	Content Validity
	Correlation
	Correlation Coefficient
	Criterion Validity
	Data Forms
	Data Plan
	Descriptive Study
	Descriptive Statistics
	Dichotomous Data
	Directional Hypothesis
	Effect Size
	Empirical Relationship
	Evaluative Study
	Evidence-Based Practice
	Excel Files for Data Analysis
	Explanatory Study
	Experimental Group Research Design
	Face Validity
	False Positive and False Negative
	Fisher Exact Test
	Frequency
	Generalization of Study Results
	GraphPad
	Group Research Designs
	One-Group Pretest-Posttest Research Design
	One-Sample t Test
	Open-Ended Questions on a Survey
	Outcome Objective
	History as a Threat to Internal Validity
	Human Service Outcome
	Hypothesis
	Hypothesis Testing
	Independent t Test
	Inferential Statistics
	Institutional Review Board
	Internal Consistency Reliability
	Interval Level of Measurement
	Limited AB Single-Subject Research Design
	Logical Generalization
	Logical Justification of a Service
	Matched Scores
	Maturation as a Threat to Internal Validity
	Mean
	Measurement Error
	Median
	Meta-Analysis of Evidence
	Measurement Error
	Median
	Mode
	Model of a Service
	Narrative Analysis of Qualitative Data
	Nominal Level of Measurement
	Null Hypothesis
	One-Group Pretest-Posttest Research Design
	One-Sample t Test
	One-Tailed Test
	Ordinal Level of Measurement
	Paired t Test
	Pie Chart
	Posttest-Only Control Group Design
	Proportion
	Pretest Scores and Posttest Scores
	Qualitative Measurement
	Quantitative Measurement
	Random Sample
	Range
	Ratio Level of Measurement
	Reliability
	Research Designs
	Research Types
	Sample
	Sampling Error
	Sample Types
	Scientific Generalization
	Scientific Justification for the Delivery of a Given Service
	Single-Subject Research Designs
	Social Desirability Bias
	Spearman Rank Correlation Coefficient
	SPSS
	Standard Deviation
	Statistics
	Statistical Significance
	Structure of the Service
	Study Population
	Systematic Review of Evidence
	Systemic Literature Review
	Systematic Random Sampling Procedure
		Test-Retest Reliability
	Threats to Internal Validity
	Treatment Group
	Treatment Period
	Two-Tailed Test
	Validity
	Variable
	Variance
Index