Research and Evaluation in Education and Psychology: Integrating Diversity With Quantitative, Qualitative, and Mixed Methods

Cover
Half Title
Series Page
Title Page
Copyright
Brief Contents
Detailed Contents
Preface
	Organization of This Book
	Pedagogical Features
	Changes in the Sixth Edition
Acknowledgments
About The Author
Introduction to Research and Evaluation
	Why Bother?
	What is research?
	Distinguishing Research and Evaluation
	Research Terminology
	Approach Taken In This Book
		Steps in the Research/Evaluation Process
		What to Expect in Chapter 1
1 - A Brief History of Research: Paradigms, Ethical Practice, and Contested Territory
	Importance of Paradigms in the History of Research
	Definition of a Paradigm
	Five Major Paradigms
		Postpositivist Paradigm
			Postpositivism: Axiology
			Postpositivist: Ontology
			Postpositivist: Epistemology
			Postpositivist: Methodology
		Constructivist Paradigm
			Constructivist: Axiology
			Constructivist: Ontology
			Constructivist: Epistemology
			Constructivist: Methodology
		Transformative Paradigm
			Why Did the Transformative Paradigm Emerge?
			Professional Associations and Guidance for Transformative Research
			Transformative Philosophical and Theoretical Basis
			Transformative: Axiology
			Transformative Ontology
			Transformative Epistemology
			Transformative Methodology
			Validity From a Transformative Perspective: A Methodological Issue
		Pragmatic Paradigm
			Pragmatic Axiology
			Pragmatic Ontology
			Pragmatic Epistemology
			Pragmatic Methodology
			Issues Related to the Pragmatic Paradigm
		Indigenous Paradigm
			Indigenous Axiology
			Indigenous Ontology
			Indigenous Epistemology
			Indigenous Methodology
	Politics, Legislation, and the Paradigms
		Why Is the Methodology of Research a Political Issue?
		Professional Organizations’ Response to NCLB
	Contested Territory: Quality, Causality, and Objectivity
	Merging Paradigms—Across Differences
	Summary of Chapter 1: An Introduction to Research
	In the Next Chapter
	notes
2 - Evaluation
	Defining Evaluation
	The Language of Evaluation
	Distinguishing Research and Evaluation
	History and Models of Evaluation
		Postpositivist Paradigm: Methods Branch
		Constructivist Paradigm: Values Branch
		Transformative Paradigm: Social Justice Branch
			Transformative Evaluation
			Culturally Responsive Evaluation
			Feminist Evaluation
			Empowerment Evaluation
		Pragmatic Paradigm: Use Branch
			Learning Organizational Evaluation
			Utilization-Focused Evaluation
			Developmental Evaluation
			Real World Evaluation
		Indigenous Paradigm: Needs and Values Branch
	Resources and Processes for Conducting Evaluations
	Steps in Planning an Evaluation
		Steps for Planning an Evaluation Study
			Focusing Stage
			Evaluation Questions
			Selection of an Evaluation Model
			Planning the Evaluation
			Management of the Evaluation
	Standards for Critically Evaluating Evaluations
		Utility
			U1 Evaluator Credibility
			U2 Attention to Stakeholders
			U3 Negotiated Purposes
			U4 Explicit Values
			U5 Relevant Information
			U6 Meaningful Processes and Products
			U7 Timely and Appropriate Communication and Reporting
			U8 Concern for Influence and Consequences
		Feasibility
			F1 Project Management
			F2 Practical Procedures
			F3 Contextual Viability
			F4 Resource Use
		Propriety
			P1 Responsive and Inclusive Orientation
			P2 Formal Agreements
			P3 Human Rights and Respect
			P4 Clarity and Fairness
			P5 Transparency and Disclosure
			P6 Conflicts of Interests
			P7 Fiscal Responsibility
		Accuracy
			A1 Trustworthy Conclusions and Decisions
			A2 Valid Information
			A3 Reliable Information
			A4 Explicit Program and Context Descriptions
			A5 Information Management
			A6 Sound Designs and Analyses
			A7 Explicit Evaluation Reasoning
			A8 Communication and Reporting
		Evaluation Accountability
			M1 Evaluation Documentation
			M2 Internal Meta-Evaluation
			M3 External Meta-Evaluation
		Multicultural Validity
	Ethics and Evaluation: the Guiding Principles
	Questions for Critically Analyzing Evaluation
		Utility
		Feasibility
		Propriety
		Accuracy
		Evaluation Accountability
		Interpersonal Validity
		Consequential Validity
		Multicultural Validity
	Evaluation Resources for Program and Policy Evaluations
		Books and Monographs
		Online Resources
		Evaluation Journals
		Professional Associations
	Summary of Chapter 2: Evaluation
	In the Next Chapter
3 - Focusing the Research: Literature Review and Community Engagement
	Reasons For Doing Literature Reviews and Engaging Community Early in the Process
		Literature Reviews and Community Engagement for Planning Primary Research
		Review of Literature as an End in Itself
	The Search Process
		Steps in the Literature Review Process
		Differences in Literature Review and Community Engagement by Paradigms
	Conducting the Literature Review Step-by-Step
		Step 1: Identify Preliminary Research Topic
			Sources of Research Topics
		Step 2: Review Secondary Sources to Get an Overview
		Step 3: Develop a Search Strategy
			Identify Preliminary Sources
			Identify Primary Research Journals
			Access Personal Networks
			Involve Community Members
		Step 4: Conduct the Search
			Prepare to Search Preliminary Sources
			What Will You See on the Computer Screen in a Search of ERIC?
			Select Titles
		Step 5: Obtain Full-Text Resources
		Step 6: Read and Prepare Bibliographic Information and Notes
			Bibliographic Information
			Notes on Each Study
		Step 7: Evaluate the Research Reports
		Step 8: Synthesize the Studies
			Narrative Synthesis
			Statistical Synthesis: Meta-Analysis
			Mixed Methods and Literature Reviews
		Step 9: Use the Literature Review
			Conceptual Framework and Program Theory
			Research Questions and Hypotheses
	Summary: Purposes of Literature Reviews
	Critical Analysis of Literature Reviews
	Questions For Critically Analyzing Literature Reviews
	Summary of Chapter 3: Literature Review and Focusing the Research
	Note
4 - Experimental and Quasi-Experimental Research
	Importance of Experimental Design
	Research Designs and Threats to Validity
		Internal Validity
		External Validity or Generalizability
		Other Threats to Validity
	Experimental, Quasi-Experimental, and Single-Group Designs
		Experimental Designs
			Pretest–Posttest Control Group Design
			Posttest-Only Control Group Design
			Single-Factor Multiple-Treatment Designs
			Solomon 4-Group Design
			Factorial Design
			Cluster Randomization Design
		Quasi-Experimental Designs
		Static-Group Comparison Design
		Nonequivalent Control Group Design
		Regression-Discontinuity (R-D) Design
		Single-Group Designs
			One-Shot Case Study
			One-Group Pretest–Posttest Design
			Time Series Design
	Other Design Issues
		Type of Treatment Variables
		Ordering Effects
		Matching
	Challenges to Using Experimental Designs in Educational and Psychological Research
	Transformative, Constructivist, and Indigenous Perspectives Regarding Experimental Research
	Postpositivist Rejoinder
	Final Thoughts
	Questions for Critically Analyzing Experimental, Quasi-Experimental, and Single-Group Designs
		Internal Validity
		External Validity (Ecological Validity)
		Other Threats to Validity
	Summary of Chapter 4: Experimental and Quasi-Experimental Research
	Where to From Here?
5 - Causal Comparative and Correlational Approaches
	Inherent Characteristics: Challenging Issues
		Focusing on Group Differences
			Sex and Gender
		Group Identification
			Race/Ethnicity
		The Myth of Homogeneity
			Indigeneity
			Disability
	Causal Comparative Research
	Correlational Research
		Steps in Conducting Correlational Research: Relationship Studies
		Graphs and Curvilinear Relationships
		Choice of a Correlation Coefficient
		Size and Interpretation
		Common or Explained Variance or r2
		Multiple Regression and Ordering Variables
		Discriminant Function Analysis
		Canonical Correlation
		Path Analysis
		Factor Analysis
		Cross-Validation
		Correlational Studies: Prediction
	Questions for Critically Analyzing Causal Comparative and Correlational Research
	Summary of Chapter 5: Causal Comparative and Correlational Research
	Where to From Here?
	Notes
6 - Survey Methods
	Design Phase
		Purposes of Surveys
		Design Considerations
		Data Collection Choices
			Surveys and Technology
		Response Rate Considerations
		Theoretical Lenses and Diverse Populations
	Sampling Plan
		Identification of the Respondents
		Population Definition
		Sampling Decisions
			Sampling Racial/Ethnic Minority Populations
			Random Digit Dialing
	Designing The Survey Instrument
		Closed-Ended Format
		Open-Ended Format
		Formatting the Questionnaire
	Pilot Testing The Questionnaire
	Language Differences Between the Researcher and the Respondent
	Special Types of Questions
		Demographic Questions
		Nonthreatening Behavioral Questions
		Sensitive Behavioral Questions
		Knowledge Questions
		Attitude Questions
	Letter of Transmittal
	Conducting the Survey
		Notes Specific to Phone Interviews
			After the Interview
		Notes Specific to Web-Based Surveys
	Data Analysis and Troubleshooting your Survey Instrument
	Questions for Critically Analyzing Survey Research
	Summary of Chapter 6: Survey Methods
	Where to From Here?
	Notes
7 - Single-Case Research
	Quality Issues In Single-Case Research: Validity and Reliability
		Quantifiable Data
		Repeated Measurements
		Direct Interventions
		Behavior of Long Duration
		Large and Immediate Impacts
		Apply to Several People
		Standardize Treatment
		Multiple Outcome Measures
		Generalization
		Multiple Components in Treatments
		Social Validation
		Reliability of Observations
	Design Options
		Phase Change Designs: Withdrawal, Reversal, or ABAB(+)
		Changing Criterion Design
		Alternating-Treatment Designs
		Multiple-Baseline Designs
			Multiple-Baseline-Across-Behaviors Design
			Multiple-Baseline-Across-People Design
			Multiple-Baseline-Across-Settings Design
			Randomization in Multiple-Baseline Designs
		Factorial Designs
	Data Analysis in Single-Case Research
		Statistical Analysis in SCD
		Practical Significance
		Improving Single-Case Designs
	Questions for Critically Analyzing Single-Case Research
	Summary of Chapter 7: Single-Case Research
	Where to From Here?
8 - Qualitative Research and Action Research
	The Qualitative Revolution in Psychology
	Basis For Selection of Qualitative Methods
		The Researcher’s View of the World
			Constructivist View
			Transformative Views
			Practical Reasons (Pragmatism)
	Types of Qualitative Research
		Ethnographic Research
		Case Study
		Phenomenological Research
		Grounded Theory
		Participatory Action Research
			Cooperative Participatory Action Research
			Transformative Participatory Action Research
	Data Collection
		Personal Interviews
			Preparing for the Interview
			Starting and Conducting the Interview
			Concluding the Interview
		Transformative Perspectives on Interviewing
			Who Can Interview Whom?
			Should the Interviewer and Interviewee Be Friends or Strangers?
			Reciprocity
			Confirmation Bias
	General Methodological Guidelines
		The Researcher Is the Instrument
		Speaking for the Other
		Focus on an Area of Inquiry
		Explore Research Sites and Sampling Issues
		Gaining Permission
		Negotiating Entry
		Entering the Field
		Role of the Researcher
		Gender Issues
	Translation Issues in Multilingual Contexts
	Qualitative Data Collection Methods
	Critically Analyzing Qualitative Research
		Credibility
		Transferability: Thick Description and Multiple Cases
		Dependability
		Confirmability
		Transformative Criteria
	Questions For Critically Analyzing Qualitative Research
	Summary of Chapter 8: Qualitative Methods
	Where to From Here?
	Notes
9 - History, Narrative Inquiry, and Autoethnography
	Why study history?
	Importance of Historical, Narrative, and Autoethnographic Research
	Types of Historical, Narrative, and Autoethnographic Research
		Sources of Data
	Steps in Conducting Historical-Narrative Research
		Step 1: Determine the Focus of the Study
			Journals as Sources of Research Problems
			“Mucking About” in an Archive or Contacting Professional Organizations
		Step 2: Conduct the Literature Review
			Preliminary Sources
		Step 3: Identify Sources of Historical Facts
			Documents and Artifacts
			Finding Information in Archives
			Accessing Historical Materials
			Using Archives
			Interviewing: Oral Histories
			Interviewing: Narrative Inquiry and Counter-Narrative
		Step 4: Synthesize and Evaluate Historical Data
			General Historical Documents
			Notes Specific to Personal Documents
		Notes on Oral History and Narrative Inquiry
	Notes On Biographical Research
	Notes on Autoethnographic, Autobiographical/Self-Study Research
	Questions for Critically Analyzing Historical-Narrative Research
	Summary of Chapter 9: History, Narrative Inquiry, and Autoethnography
	Where to From Here?
	Historiographic Texts, Journals, and Other Resources
		Feminist/Racial/Ethnic/LGBTQ+ Resources
	Electronic Resources
	Useful Journals of Historical-Narrative Research
	Notes
10 - Mixed Methods Research
	Definition and Characteristics
	Importance in Educational and Psychological Research
	Philosophical Assumptions and Mixed Methods Research
	Mixed Methods Design Options
		Pragmatic Mixed Methods Design
		Postpositivist Mixed Methods Design
		Constructivist Mixed Methods Design
		Transformative Mixed Methods Design
		Indigenous Mixed Methods Designs
	Questions for Critically Analyzing Mixed Methods Research
	Summary of Chapter 10: Mixed Methods Research
	Where to From Here?
	Notes
11 - Sampling
	Definition, Selection, and Ethics
		Sampling Strategies: Alternative Paradigms
	External Validity (Generalizability) or Transferability
	Defining the Population and Sample
	Identification of Sample Members
		Identification of Race and Ethnicity in Populations
		People With Disabilities
	Sampling Strategies: Dimensions of Diversity
	Sampling Strategies: Types of Sampling
		Probability-Based Sampling
			Simple Random Sampling
			Systematic Sampling
			Stratified Sampling
			Cluster Sampling
			Multistage Sampling
			Complex Sampling Designs in Quantitative Research
		Examples of Sampling in Quantitative Studies
		Purposeful and Theoretical Sampling
			Extreme or Deviant Cases
			Intensity Sampling
			Maximum-Variation Sampling
			Homogeneous Sampling
			Typical-Case Sampling
			Stratified Purposeful Sampling
			Critical-Case Sampling
			Snowball or Chain Sampling
			Criterion Sampling
			Theoretical Sampling
			Confirming and Disconfirming Cases
			Opportunistic Sampling
			Purposeful Random Sampling
			Sampling Politically Important Cases
			Case Study Sampling
		Examples of Qualitative Research Sampling
		Convenience Sampling
		Mixed Methods Sampling
	Sampling Bias
	Access Issues
		Access in Educational Settings
		Access to Records
	Sample Size
		Rules of Thumb
			Quantitative Research Rules of Thumb
			Qualitative Research Rules of Thumb
		Formulaic Determinations of Sample Size
	Ethics and Protection of Study Participants
		Informed Consent
		Marginalized Populations and Informed Consent
		Confidentiality and Anonymity
		Deception in Research Studies
	Questions for Critically Analyzing Sampling Strategies
	Summary of Chapter 11: Sampling
12 - Data Collection
	Operationalizing Concepts
	Quantitative Measurement Topics
		Data Collection and Technology
		Performance and Portfolio Assessment
		Testing and Technology
		Secondary Data Sources
	Selection and Development of Quantitative Instruments
		Identifying an Instrument: Sources of Information
		Advice for Selecting an Instrument
	Developing a Data Collection Instrument
		Step 1. Define the Objective of Your Instrument
		Step 2. Identify the Intended Respondents and Make Format Decisions
		Step 3. Review Existing Measures
		Step 4. Develop an Item Pool
			Cultural Responsiveness in Instrument Development
		Step 5. Prepare and Pilot Test the Prototype
		Step 6. Conduct an Item Analysis and Revise the Measure
		Advice for Administering Data Collection Instruments
	Qualitative Data Collection Methods
		Qualitative Data Collection and Mobile Devices
		Observation
		Group and Individual Interviews and Technology
			Individual Interviews
		Interviews and Technology
			Recording Interviews
			Discussion Board Qualitative Data Collection
			Focus Groups
			Example of Focus Group Questions
		Document and Records Review and Technology
		Visual Data and Technology
		Participatory Data Collection Strategies
			Participatory Rural Appraisal
			Beneficiary Assessment
			Self-Esteem, Associative Strength, Resourcefulness, Action Planning, and Responsibility
			Appreciative Inquiry
		Indigenous Data Collection
	Mixed Methods and Data Collection
	Standards for Judging Quality of Data Collection
		Postpositivist: Reliability
			Repeated Measures Reliability
			Internal Consistency
			Reliability With Observers
		Constructivist: Dependability
		Postpositivist: A Unified Concept of Validity
			Sources of Evidence for Construct Validity
			Evidence Based on Response Patterns
			Evidence Based on Relationship of Test Scores With Other Variables
			Evidence Based on the Internal Structure of the Test
			Evidence Based on Test Content
			Evidence Based on Test–Criterion Relationship
			Evidence for Validity and the Consequences of Testing
		Constructivist: Credibility
		Postpositivist: Objectivity
		Constructivist: Confirmability
		Transformative Paradigm: Avoidance of Bias
		Universal Design and Instrument Development
			Indigenous: Avoidance of Bias
	Questions or Critically Analyzing Data Collection
		Methodological Validity
	Summary of Chapter 12: Data Collection
	Educational and Psychological Measurement Resources
	Measurement Journals
	Websites
13 - Data Analysis, Interpretation, and Use
	Quantitative Analysis Strategies
		Commonly Used Quantitative Data Analysis Techniques
			Scale of Measurement
			Descriptive Statistics
			Correlational Statistics
			Statistical Significance
			Inferential Statistics
			Variations on ANOVA
			Post Hoc Analysis
		Choice of Statistic
			Assumptions for Parametric Statistics
		Computers and Quantitative Analysis
		Interpretation Issues in Quantitative Analysis
			Randomization
			Sample Size
			Statistical Versus Practical Significance
			Cultural Bias and Engagement With Stakeholders in Interpretation
			Generalizability
			Options for Reporting Statistical Results
			Replication
			Use of Nonparametric Statistics
			Competing Explanations
			Recognizing Limitations
			Principled Discovery
	Qualitative Analytic Strategies
		Steps in Qualitative Data Analysis
			Step 1: Preparing the Data for Analysis
			Steps 2 and 3: Data Exploration and Data Reduction
		Additional Thoughts on Qualitative Data Analysis
		Theoretical Lenses and Qualitative Data Analysis
		Using Computers in Qualitative Analysis
		Interpretation Issues in Qualitative Data Analysis
			Triangulating Data
			Audits
			Cultural Bias
			Generalization/Transferability
			Member Checks
	Analytic and Interpretive Issues in Mixed Methods
	A Research Plan: The Management Plan and Budget
		The Management Plan
		The Research Budget
	Writing Research Reports
		Writing Reports
		Dissertations and Theses
		Reporting Format Options
			Quantitative Reports
		Qualitative Reports
		Digital Dissemination
		Clarity in Writing
	Utilization of the Research Results
		Writing for Publication
		Submitting a Manuscript
		Use for Social Change
	Questions for Critically Analyzing Data Analysis and Interpretation
		Quantitative Research
		Qualitative Research
		Mixed Methods (these criteria overlap with those presented in Chapter 10)
		Interpretation Issues
	Summary of Chapter 13: Data Analysis, Interpretation, and Use
	notes
Appendix
	Getting the Research Idea
	Research Concept Paper
	Formal Research Proposal
	Proposal Considerations
		Chapter 1: Focus of the Study
		Chapter 2: Literature Review
		Chapter 3: Methodology
		Recommended Proposal Document Format
References