Judgment in Predictive Analytics

Preface
	Reference
Acknowledgments
Contents
Part I: Judgment in Human-Machine Interactions
	Chapter 1: Beyond Algorithm Aversion in Human-Machine Decision-Making
		1 Introduction
		2 The Human vs. Machine Debate in Judgment and Decision-Making
		3 Human-Machine Decision-Making
		4 Beyond Algorithm Aversion: What Is Algorithm Misuse?
		5 Causes of Algorithm Aversion and Algorithm Misuse
			5.1 Prior Knowledge
			5.2 Decision Control
			5.3 Incentive Structures
			5.4 Alignment of Decision-Making Processes
			5.5 Alignment of Decision-Making Objectives
		6 Towards Improved Methods and Metrics for Understanding and Resolving Algorithm Misuse
		7 Conclusion
		References
	Chapter 2: Subjective Decisions in Developing Augmented Intelligence
		1 Introduction
		2 Theoretical Framework
			2.1 Machine Learning-Based Augmented Reality
			2.2 Design Science
			2.3 Decision Making
		3 Development Process
			3.1 Use Case: Finding a Starting Point
			3.2 MVP: Getting a First Version
				3.2.1 Camera Feed
				3.2.2 Execution Engine and Detection Model
				3.2.3 Image Processing
				3.2.4 Visualization
			3.3 Summary of Steps 3-7: From a Proof of Concept to Future Use Cases
		4 Decisions and Heuristics During the Development Process
			4.1 Decision Types
				4.1.1 Framework Decisions
				4.1.2 Technological Decisions
				4.1.3 Design Decisions
			4.2 Decision Pyramids
				4.2.1 Successive Decisions
				4.2.2 Small and Large Worlds
			4.3 Exemplary Development Decisions
				4.3.1 General Environment
				4.3.2 Framework Decisions
				4.3.3 Technological Decisions
				4.3.4 Design Decisions
		5 Discussion
		6 Limitations and Outlook
		References
	Chapter 3: Judgmental Selection of Forecasting Models (Reprint)
		1 Introduction
		2 Literature
			2.1 Commonly Used Forecasting Models
			2.2 Algorithmic Model Selection
			2.3 Model Selection and Judgment
			2.4 Combination and Aggregation
		3 Design of the Behavioral Experiment
			3.1 Selecting Models Judgmentally
			3.2 Data
			3.3 Participants
			3.4 The Process of the Experiment
			3.5 Measuring Forecasting Performance
		4 Analysis
			4.1 Individuals´ Performance
			4.2 Effects of Individuals´ Skill and Time Series Properties
			4.3 50% Statistics + 50% Judgment
			4.4 Wisdom of Crowds
			4.5 Evaluation Summary and Discussion
		5 Implications for Theory, Practice, and Implementation
		6 Conclusions
		Appendix
			Forecasting Models
			Participants Details
		References
	Chapter 4: Effective Judgmental Forecasting in the Context of Fashion Products (Reprint)
		1 Introduction
		2 Theoretical Background
			2.1 Judgment Analysis
			2.2 Forecasting the Demand of Fashion Products
			2.3 Hypotheses
		3 Methods
		4 Empirical Setting
		5 Results
		6 Discussion
		References
	Chapter 5: Judgmental Interventions and Behavioral Change
		1 Background
		2 The Design of a Behavioral Experiment
		3 Results
		4 Discussion
		5 Conclusions
		References
Part II: Judgment in Collective Forecasting
	Chapter 6: Talent Spotting in Crowd Prediction
		1 Introduction
			1.1 Definition of Skill
			1.2 Five Categories of Skill Correlates
		2 Study 1
			2.1 Study 1: Methods
				2.1.1 Literature Search
				2.1.2 Outcome Variables
				2.1.3 Predictors of Skill
					2.1.3.1 Accuracy-Related
					2.1.3.2 Intersubjective
					2.1.3.3 Behavioral
					2.1.3.4 Dispositional
						Fluid Intelligence and Related Measures
					2.1.3.5 Expertise-Related
			2.2 Study 1: Results
				2.2.1 Accuracy-Related
				2.2.2 Intersubjective
				2.2.3 Behavioral
				2.2.4 Dispositional
				2.2.5 Expertise-Related
			2.3 Study 1 Discussion
		3 Study 2
			3.1 Study 2: Methods
				3.1.1 Good Judgment Project Data
				3.1.2 Cross-Validation and Outcome Variable Definition
				3.1.3 Predictor Selection
				3.1.4 Statistical Tests
			3.2 Study 2: Results
				3.2.1 Correlational Analyses
					3.2.1.1 Accuracy-Related Measures
					3.2.1.2 Intersubjective Measures
					3.2.1.3 Behavioral Measures
					3.2.1.4 Dispositional Measures
					3.2.1.5 Expertise Measures
				3.2.2 Multivariate LASSO Models
			3.3 Study 2: Discussion
		4 General Discussion
			4.1 Research Synthesis
			4.2 Use Cases
			4.3 Limitations and Future Directions
			4.4 Conclusion
		Appendix: Methodological Details of Selected Predictors
			Item Response Theory Models
			Contribution Scores
		References
	Chapter 7: Performance-Weighted Aggregation: Ferreting Out Wisdom Within the Crowd
		1 Introduction
			1.1 The Wisdom of Crowds
			1.2 Judgment Quality: Defining and Identifying Expertise in the Crowd
		2 Judgment Aggregation Strategies
			2.1 Mean Strategies
			2.2 Median Strategies
			2.3 Weighting Functions
				2.3.1 Weight All
				2.3.2 Select Crowd
				2.3.3 Hybrid Weighting Functions
			2.4 Choosing a Weighting Function
		3 Indicators of Expertise
			3.1 History-Based Methods
				3.1.1 Cooke´s Classical Method
				3.1.2 Contribution Weighted Model
				3.1.3 Discussion
			3.2 Disposition-Based Methods
				3.2.1 Domain Expertise
				3.2.2 Psychometric Indicators of Individual Differences
				3.2.3 Discussion
			3.3 Coherence-Based Methods
				3.3.1 Coherence Approximation Principle
				3.3.2 Probabilistic Coherence Scale
				3.3.3 Discussion
		4 General Discussion
			4.1 Ensemble Methods
			4.2 Conclusion
		References
	Chapter 8: The Wisdom of Timely Crowds
		1 Introduction
			1.1 Forecaster Evaluation
			1.2 Time Decay
			1.3 Time and Crowd Size
		2 Evaluating Forecasters Over Time
			2.1 Forecast Timing
			2.2 Information Accrual
			2.3 Reliability of Forecaster Assessment
			2.4 Recommendations
		3 The Timeliness of Crowds
			3.1 Selection Methods
			3.2 Weighting Methods
			3.3 Comparing Methods
			3.4 A Probabilistic Hybrid Method
			3.5 Martingale Violations
			3.6 Recommendations
		4 Crowd Size and Timing
			4.1 Resampling the Crowd
		5 General Discussion
			5.1 Signal Sources
			5.2 Bias
			5.3 Beyond Judgmental Forecasting
			5.4 Summary of Recommendations
				5.4.1 Evaluating Forecasters
				5.4.2 Information Accrual
				5.4.3 Forecast Recency and Aggregation
				5.4.4 Time and Crowd Size
		References
Part III: Contextual Factors and Judgmental Performance
	Chapter 9: Supporting Judgment in Predictive Analytics: Scenarios and Judgmental Forecasts
		1 Introduction
		2 Literature Review
		3 Methodology
			3.1 Experimental Design
				3.1.1 Phase 1: Individual Forecasts
				3.1.2 Phase 2: Team Forecasts with Scenario Discussions
				3.1.3 Phase 3: Final/Preferred Individual Forecasts After Scenario Discussions
			3.2 Results
				3.2.1 Assessments of Scenario Tone
				3.2.2 Individual Forecasts
				3.2.3 Team Forecasts with Scenario Discussions
				3.2.4 Final/Preferred Individual Forecasts After Scenario Discussions
		4 Discussion
		5 Conclusion
		References
	Chapter 10: Incorporating External Factors into Time Series Forecasts
		1 Introduction
		2 External Events
			2.1 Event Characteristics
				2.1.1 Magnitude and Duration
				2.1.2 Regularity and Frequency
				2.1.3 Predictability
			2.2 Event Impact
				2.2.1 Magnitude
				2.2.2 Direction
				2.2.3 Duration
				2.2.4 Type
		3 The Role of Judgment in Dealing with External Events
			3.1 Judgmental Adjustment of Statistical Forecasts from Series Disrupted by External Events
			3.2 Using Judgment to Select and Clean Data to Produce Baseline Forecasts
			3.3 Judges´ Use of Analogical Strategies to Make Forecasts When Series Are Disrupted by External Events
		4 Statistics to the Rescue?
			4.1 Non-transparent Models
			4.2 Transparent Models
		5 Summary
		References
	Chapter 11: Forecasting in Organizations: Reinterpreting Collective Judgment Through Mindful Organizing
		1 Introduction: Slow Progress Behind Paradigmatic Blinkers?
		2 Showcasing the Effects of Functionalism in Forecasting Research
			2.1 Extracting Forecasts from Groups
			2.2 Learning from Feedback
		3 Nuanced Organizational Aspects Towards a New Framework in Forecasting
			3.1 Learning from Success Versus Failure
			3.2 Group Deliberation About Performance
			3.3 Team Leaders as Facilitators
		4 Mindful Organizing: A Framework in the Interpretivist-Functionalist Transition Zone
		5 Inducing Mindful Organizing to Debias Group Judgment
			5.1 Focus on Episodic, Dramatic Error
			5.2 Use of Analogical Reasoning and Reference Classes
		6 Conclusion
		References
Index