Introduction to Theory-Driven Program Evaluation: Culturally Responsive and Strengths-Focused Applications

Cover
Half Title
Title Page
Copyright Page
Dedication
Contents
Preface
Acknowledgments
Part I: Foundations and Strategies for Theory-Driven Program Evaluation
1. The Evolution of Program Theory-Driven Evaluation Science
	Comprehensive Theory of Evaluation Practice
		History of Evaluation Theory
		Looking Toward the Future
		Refinements for Contemporary Evaluation Practice
		Defining Program Theory-Driven Evaluation Science
	Potential Benefits of Theory-Driven Evaluation Science
	Toward a Science of Developing People and Organizations
		Human Transformation Technologies
		Organizational Development
			Improving Organizational Members
			Improving Organizational Effectiveness
			Applications
	Purpose and Organization of This Book
2. Developing Program Impact Theory
	Efficacy versus Effectiveness Evaluation
	Conceptual Framing
	The Content of Program Theory
	Direct Effect
		Mediation Models
		Path Moderators
	The Process of Developing Program Impact Theory
		Engage Relevant Stakeholders
		Develop a First Draft
		Present First Draft to Stakeholders
		Plausibility Check
		Probe Arrows for Model Specificity
			Finalize Program Impact Theory (Step I)
	Conclusion
3. Formulating, Prioritizing, and Answering Evaluation Questions with Credible and Actionable Evidence
	Formulating Evaluation Questions
	Prioritizing Evaluation Questions
	Answering Evaluation Questions
	Practicing Through Class Projects
	Conclusions
Part II: Culturally Responsive and Strengths-Focused Applications
4. Designing and Redesigning Programs in a Global Pandemic: Following the Theory-Driven Evaluation Science
	Follow the Science
	Following the Theory-Driven Evaluation Science to Promote Well-Being
	What Can We Learn from Positive Psychological Science?
	Designing and Redesigning Programs in the Global Pandemic
	Searching for Successful Evidence-Based PPIs
	Conclusions
5. Culturally Responsive and Strengths-Focused Evaluation Practice Adventures
	Pursuing Evidence for Strengths Across Cultures
	Cross-Cultural Evaluation Capacity Building
	Formative Evaluation Across National Boarders
	Exemplary Evaluation in a Multicultural World
	Published Theory-Driven Program Evaluations
	Conclusions
6. The Evaluation of a $20 Million, Multiprogram, Work and Health Initiative
	Promoting Health Through Work
	Mission of WHI
	Evaluation Approach and Initiative Framework
	Data Collection
	Formative Evaluation Tools
	Summative Evaluation
	Conclusion
7. Evaluation of the Winning New Jobs Program
	Developing a Program Impact Theory for WNJ
		Program Description and Rationale
		Los Angeles County Office of Education
		NOVA Private Industry Council
		Proteus, Inc.
		Program Impact Theory Development
		Formulating and Prioritizing Evaluation Questions
		Answering Evaluation Questions
		Data Collection
		Key Evaluation Findings and Conclusions
		Summary Answer to Evaluation Question Number I—Program Implementation
			Supporting Evidence for Answering Evaluation Question Number 1
		Summary Answer to Evaluation Question Number 2--Program Service
			Supporting Evidence for Answering Evaluation Question Number 2
		Forty Percent Failed to Meet One or More of the Original Eligibility Criteria
		Summary Answer to Evaluation Question Number 3—Short-Term Outcomes
			Supporting Evidence for Answering Evaluation Question Number 3
		Evaluation Design Limitations
		Summary Answer to Evaluation Question Number 4—Reemployment Outcomes
			Supporting Evidence for Answering Evaluation Question Number 4
		Response Rates
		Reemployment Rates
		Summary Answer to Evaluation Question Number 5—Program Sustainability and Replication
			Supporting Evidence for Answering Evaluation Question Number 5
		Dissemination Efforts
		Program Replication
	Considerations for Future Programming
		Workforce Preparation and Basic Skill Levels
		Removing Additional Employment Obstacles Prior to WNJ
		Clarifying Program Sustainability and Replication Expectations
		Demonstrating WNJ Is Better Than Existing Programs
	Conclusion
8. Evaluation of the Computers in Our Future Program
	Developing a Program Impact Theory for CIOF
		Program Description and Rationale
		Developing Program Impact Theory
		Formulating and Prioritizing Evaluation Questions
		Answering Evaluation Questions
		Data Collection
		Key Evaluation Findings and Conclusions
			1. Program Implementation
			2. Program Service
			Computer Experience and Usage
			Characteristics of Population Served
			3. Program Impact
			Knowledge and Skill Gains
			Technical Skill Gains at Follow-Up
			Employment Status
			Employment Outcomes
			Educational Status
			Educational Benefits
			Personal Benefits
			Attitudes toward Computers
			New Computer Knowledge and Skills
			Employment-Related Benefits
			Use and Application of Technology
			Changed Outlook and/or Aspirations
			What Center Users Liked Most
			Center Environment
			Staff and Instructors
			Access to Computers
			Structure of the Centers
			Educational Benefits
			CIOF as a Community Resource
			Staff and Instructors
			Access to Center and Computers
			Internet Access
			Access to More and Better Programs
			Warm Center Environment
			Other Key Resources Developed
			4. Strengths and Weaknesses of the CIOF Model
			Technology Education
			Employment Linkages
			Policy Development
			Resource Procurement
			Considerations for Future Programming
			Professional Program Planning and Development Assistance
			Facilitate Strong Program Leadership
			Provide Core Curriculum Modules
	Conclusion
9. Evaluation of the Health Insurance Policy Program
	Developing a Program Impact Theory for Hipp
		Program Description and Rationale
		Program Impact Theory Development
		Formulating and Prioritizing Evaluation Questions
	Answering Evaluation Questions
		Data Collection
		Key Evaluation Findings and Conclusions
			Publications Development
			More Policy Analysis Desired by Targeted Constituents
			Coordination and Integration of Advisory Information
			Report Dissemination and Follow-up
			Dissemination Strategy
			Dissemination Targets
			Initial Dissemination Strategy
			Technical Assistance with Dissemination Efforts Were Needed
			Coordination Challenges
			Media Coverage
		Research and Policy Recommendations
		Research Outcomes
			Response Rates
		Awareness
		Understanding
			Most Informative Sections
			Key Messages Taken Away
		Influence
			Perceived Credibility of Reports
			Usefulness of HIPP Research
			How HIPP Reports Were Used by Respondents
			Recommendations for Improvement
		Considerations for Future Programming
			Comprehensive Reports Were More Effective than Policy Alerts
			Media and Communications Technical Assistance was Crucial
			Productive Research Collaborations across Universities were Difficult to Sustain
	Conclusion
10. Evaluation of the Future of Work and Health Program
	Developing a Program Impact Theory for the FWH Program
		Program Description and Rationale
		Modification of Original Program Goals
		Key Challenges with New Overarching Objectives
			FWH Program Was Reconfigured Again
		Program Impact Theory Development
	Formulating and Prioritizing Evaluation Questions
	Answering Evaluation Questions
		Data Collection
		Key Evaluation Findings and Conclusions
			Identifying Trends
			Research Projects
			Issues and Trends Examined
			Research Topics Explored Within the Longitudinal CWHS Dataset
			Key Trends Identified
			Key Trends Identified by California Work and Health Survey
			Knowledge Gained About the Three Trends
		Building a Network
			Conference Utility for Building Contacts and Networking
		Building Knowledge
			Effectiveness in Identifying Work and Health Connections
			Usefulness of Knowledge Gained
			Work and Health Connections Not Covered
		Identifying Policy
			California Center for Health Improvement (CCHI)
			SB910
			San Francisco Board of Supervisors' Finance Committee
			Latino Issues Forum, San Francisco
			Employee Benefits Research Institute (EBRI) Forum on Health Insurance
			Working Partnerships
			Usefulness of Conference in Identifying Policy Implications
		Dissemination
			Media Coverage
			Research Reports
	Considerations for Future Programming
		Need for Direct Links between Program Goals and Research Activities
		Need for Mechanisms for Integrating and Synthesizing Findings
		The Employer Perspective was Missing
		Link Policy and Dissemination with Research Efforts
		Require Key Grantees to Participate in Evaluation
	Conclusion
11. Evaluation of the Entire Work and Health Initiative
	Evaluation of the Entire Initiative
	Crosscutting Evaluation Findings and Conclusions
		Understanding Links between Work and Health
		Increasing High Quality Employment
		Improving Conditions of Work
			Expand the Availability of Worksite Health Programs and Benefits
			Evaluation of Areas of Special Interest to TCWF
			Capacity Building
			Effectiveness of Program Coordination Teams
			Sustainability
			Effectiveness of Initiative Program Management
	Considerations for Future Initiatives and Grantmaking
	Conclusion
12. The Case: Bunche-Da Vinci Learning Partnership Academy
	The Community
	The School
	A New Principal
	Challenges and Unfinished Business
	Note
13. Using Program Theory-Driven Evaluation Science to Crack the Da Vinci Code
	Cracking the "Da Vinci Code"
	Negotiating a Realistic and Fair Contract
		Background Rationale
	Evaluation Plan
		Bunche-Da Vinci Realistic Job Preview
	Reflections and Conclusions
	Note
Part III: Reflections, Lessons Learned, and Future Directions
14. Reflections and Lessons Learned
	Reflections and Lessons Learned from Recent Applications
		Simple to Complex Roadmaps for Evaluation Practice
		Recognizing the Influence of Culture
		The Value of Added of Strengths-Focused Questions
		The Challenges of Gathering Credible and Actionable Evidence
	Reflections and Lessons Learned From the WHI Evaluation
		Evaluation of Winning New Jobs
		Developing Program Impact Theory for WNJ
		Formulating and Prioritizing Evaluation Questions for WNJ
		Answering Evaluation Questions for WNJ
	Evaluation of Computers in Our Future
		Developing Program Impact Theory for CIOF
		Formulating and Prioritizing Evaluation Questions for CIOF
		Answering Evaluation Questions
	Evaluation of the Health Insurance Policy Program
		Developing Program Impact Theory for HIPP
		Formulating and Prioritizing Evaluation Questions for HIPP
		Answering Evaluation Questions
	Evaluation of the Future of Work and Health Program
	Lessons Learned Across the Evaluations of the WHI
		Interactive, Bidirectional Program Impact Theory Development was Most Effective
		Reaching Consensus Was NOT Difficult
		Complexity Is the Enemy: Strive for Parsimonious Program Impact Theories
		Unpack Program Process Theory Selectively
		Potential for Substantial Time and Resource Savings
		Program Implementation Improvement without Evaluation
		Formative Evaluation Can Lead to Goal Change Instead of Program Improvement
		Strategies for Managing Evaluation Anxiety Are Often Required
		Continuous Improvement Feedback: Don't Forget the Positive Findings
		Building Program and Evaluation Capacity
		Managing the Formative to Summative Evaluation Transition
		Credible Evidence: Often the Achilles Heel of Effectiveness Evaluation
	Reflections on the Bunche-Da Vinci Proposal
	Links to Efficacy Evaluations
	Conclusion
15. Future Directions for the Transdiscipline and Profession of Evaluation Science
	Evaluation Science: An Emerging Transdiscipline
		Using Evaluation Theory in Practice
		Developing an Evidence-Based Evaluation Practice
	Implications for the Future Theory-Driven Program Evaluation
	Conclusion
References
	About The Author
Index