Common mistakes in meta-analysis and how to avoid them

Contents
1. Preface
2. Acknowledgements
3. Workshops on Meta-Analysis
4. Web Sites
	4.1. Web site for this book
	4.2. Web site for the software
	4.3. Web site for our workshops
5. How to read the plot
6. Statistical models for meta-analysis
	6.1. Overview
		6.1.1. Random effects
		6.1.2. Fixed effect (singular)
		6.1.3. Fixed effects (plural)
		6.1.4. Three textbook cases
		6.1.5. Random effects
		6.1.6. Fixed effect (singular)
		6.1.7. Fixed effects (plural)
	6.2. Each model is appropriate for a specific inference
		6.2.1. How the model affects the confidence-interval width
		6.2.2. How the model affects the estimate of the mean effect size
		6.2.3. How the model affects our ability to address heterogeneity
		6.2.4. How the model affects the meaning of the null hypothesis
7. Mistakes in choosing a statistical model
	7.1. Overview
	7.2. Choosing between fixed effect (singular) and random effects
		7.2.1. Mistake
		7.2.2. Details
		7.2.3. Cases where the fixed-effect (singular) model applies
		7.2.4. Cases where the random-effects model applies
		7.2.5. In context
		7.2.6. What difference does it make?
		7.2.7. Examples
		7.2.8. Example | PTSD in parents of children with chronic illness
		7.2.9. Example | Preoperative statin therapy
		7.2.10. The correct approach
		7.2.11. Example | Interventions to promote physical activity
		7.2.12. Example | Dropout rate in adult psychotherapy
		7.2.13. Example | Impact of preference in psychotherapy
		7.2.14. Example | Behavior-change techniques for asthma patients
		7.2.15. Example | Emotional congruence with children
		7.2.16. Testing for heterogeneity when the analysis is based on one population
		7.2.17. Constraints of the fixed-effect model
	7.3. Choosing between fixed effects (plural) and random effects
		7.3.1. Mistake
		7.3.2. Details
		7.3.3. The fixed-effect model
		7.3.4. The fixed-effects model
		7.3.5. Where the fixed-effects model applies
		7.3.6. When studies are pulled from the literature
	7.4. Limitations of the random-effects model
		7.4.1. Mistake
		7.4.2. Details
		7.4.3. Assumptions of the random-effects model
		7.4.4. A textbook case
		7.4.5. When studies are pulled from the literature
		7.4.6. A useful fiction
		7.4.7. Transparency
		7.4.8. A narrowly defined universe
		7.4.9. In context
		7.4.10. Extreme cases
	7.5. Knapp-Hartung adjustment
		7.5.1. Mistake
		7.5.2. Details
		7.5.3. Limitations of the Knapp-Hartung adjustment
	7.6. Meta-analysis in legal applications
		7.6.1. Mistake
		7.6.2. Details
		7.6.3. Fixed-effect model (singular) vs. fixed-effects model (plural)
	7.7. Putting it all together
Summary
8. Issues and myths about statistical models
	8.1. Overview
	8.2. Random-effects model assigns equal weight to all studies
		8.2.1. Mistake
		8.2.2. Details
	8.3. Random-effects model gives too much weight to small studies
		8.3.1. Mistake
		8.3.2. Details
		8.3.3. Example | Impact of GLP-1 mimetics on blood pressure
		8.3.4. Study quality vs. risk of bias
	8.4. Comparing results from the two models
		8.4.1. Mistake
		8.4.2. Details
		8.4.3. Example | Impact of educational programs
	8.5. Random-effects model is more conservative
		8.5.1. Mistake
		8.5.2. Details
		8.5.3. Example | Statin use and bladder cancer
		8.5.4. Failure to reject the null hypothesis may not be conservative
		8.5.5. Random-effects model may not be conservative
		8.5.6. Example | Water chlorination and cancer
	8.6. Fixed-effect model has better statistical power
		8.6.1. Mistake
		8.6.2. Details
	8.7. When τ2 is estimated as zero
		8.7.1. Mistake
		8.7.2. Details
		8.7.3. What difference does it make?
		8.7.4. Example | High-dose vs. standard-dose statins
		8.7.5. Example | Volunteer tutoring programs
	8.8. Switching models will have major impact on results
		8.8.1. Mistake
		8.8.2. Details
	8.9. Meta-analyses with large N will have good power
		8.9.1. Mistake
		8.9.2. Details
	8.10. Putting it all together
Summary
Summary
Summary
Summary
Summary
Summary
9. Heterogeneity
	9.1. Overview
		9.1.1. What do we mean by heterogeneity?
		9.1.2. Heterogeneity in a primary study
		9.1.3. Heterogeneity in a meta-analysis
		9.1.4. The sources of confusion
	9.2. Heterogeneity is bad
		9.2.1. Mistake
		9.2.2. Details
		9.2.3. Heterogeneity affects what we can learn from the analysis
		9.2.4. The good folks of New Cuyama
	9.3. The prediction interval
		9.3.1. Mistake
		9.3.2. Details
		9.3.3. Example | Effect of methylphenidate on cognitive function in adults with ADHD
		9.3.4. Example | Impact of GLP-1 mimetics on blood pressure
		9.3.5. When τ2 is estimated as zero
		9.3.6. Example | High dose vs. standard dose of statins
		9.3.7. Computing prediction intervals
		9.3.8. Some caveats regarding the prediction interval
		9.3.9. The prediction interval is only a first step
		9.3.10. The normal curve
		9.3.11. Reliability of the prediction interval
	9.4. Prediction interval vs. confidence interval
		9.4.1. Mistake
		9.4.2. Details
		9.4.3. Example | Prevalence of ADHD in patients with SUD
		9.4.4. Example | Augmenting clozapine with a second antipsychotic
		9.4.5. Example | Impact of GLP-1 mimetics on blood pressure
		9.4.6. Impact of additional studies
		9.4.7. Formulas
		9.4.8. Future options
	9.5. Mistakes in using the I2 statistic
		9.5.1. Mistake
		9.5.2. Details
		9.5.3. Examples using the standardized mean difference
		9.5.4. Examples using risk ratios
		9.5.5. Words matter
		9.5.6. Example | Drugs for ADHD
		9.5.7. Example | Exercise for chronic back pain
		9.5.8. In context
		9.5.9. Using the I2 statistic correctly
		9.5.10. Further readings
	9.6. Classifying heterogeneity as low, moderate or high
		9.6.1. Mistake
		9.6.2. Details
		9.6.3. Example | Allegiance to treatment
		9.6.4. Example | Prevalence of pelvic-floor disorders
		9.6.5. Example | Preventing substance abuse
		9.6.6. Example | Exercises for back pain
		9.6.7. In context
	9.7. Using the p-value as index of heterogeneity
		9.7.1. Mistake
		9.7.2. Details
		9.7.3. Example | Impact of preoperative statin therapy
		9.7.4. Example | Impact of smoke-free legislation
	9.8. Using the Q-value as index of heterogeneity
		9.8.1. Mistake
		9.8.2. Details
		9.8.3. Example | Impact of preoperative statin therapy
		9.8.4. Example | Impact of smoke-free legislation
		9.8.5. Q does tell us one thing about the dispersion
	9.9. Estimates of variance may not be reliable
		9.9.1. Mistake
		9.9.2. Details
	9.10. Statistics for heterogeneity refer to fixed-effect model
		9.10.1. Mistake
		9.10.2. Details
		9.10.3. Example | Serotonin-Aggression relation
	9.11. Putting it all together
Summary
Summary
Summary
Summary
Summary
Summary
Summary
10. Mistakes related to significance testing
	10.1. Overview
		10.1.1. NHST vs. effect-size estimation in primary studies
		10.1.2. Cases where NHST is the preferred approach
		10.1.3. Cases where effect-size estimation is the preferred approach
		10.1.4. Meta-analysis
	10.2. When the effect size is consistent across studies
		10.2.1. Mistake
		10.2.2. Details
		10.2.3. Example | Tamiflu
	10.3. When the effect size varies across studies
		10.3.1. Mistake
		10.3.2. Details
		10.3.3. The null hypothesis may not apply to any specific population
		10.3.4. The null hypothesis applies to a specific mix of populations
		10.3.5. Example | ADHD
		10.3.6. Example | Clozapine
		10.3.7. Example | Juvenile Drug Courts
		10.3.8. In context
	10.4. Significant effect may be  harmful in some populations
		10.4.1. Mistake
		10.4.2. Details
	10.5. Putting it all together
Summary
Summary
Summary
11. Publication Bias
	11.1. Overview
		11.1.1. Example | Second-hand smoke and lung cancer
	11.2. Conflating bias with the small-study effect
		11.2.1. Mistake
		11.2.2. Details
		11.2.3. Use logic in trying to disentangle bias from small study
	11.3. Publication bias does not invalidate the analysis
		11.3.1. Mistake
		11.3.2. Details
	11.4. Tests to detect bias may be over-interpreted
		11.4.1. Mistake
		11.4.2. Details
	11.5. Trim and fill
		11.5.1. Mistake
		11.5.2. Details
		11.5.3. Additional problems associated with Trim and Fill
	11.6. The tests only work under certain conditions
		11.6.1. Mistake
		11.6.2. Details
	11.7. Procedures do not apply to studies of prevalence
		11.7.1. Mistake
		11.7.2. Details
	11.8. The model for publication bias is simplistic
		11.8.1. Mistake
		11.8.2. Details
	11.9. Publication bias and the grey literature
		11.9.1. Mistake
		11.9.2. Details
	11.10.   Lines on funnel plot
		11.10.1. Mistake
		11.10.2. Details
	11.11. Fail-Safe N
		11.11.1. Mistake
		11.11.2. Details
	11.12.   Using cumulative analysis
		11.12.1. Mistake
		11.12.2. Details
	11.13.   The focus on publication bias ignores other types of bias
		11.13.1. Mistake
		11.13.2. Details
	11.14. Putting it all together
Summary
Summary
Summary
Summary
Summary
Summary
Summary
Summary
Summary
12. Mistakes in subgroup analyses
	12.1. Overview
		12.1.1. Example | Drugs for weight-loss
	12.2. Assuming a causal relationship
		12.2.1. Mistake
		12.2.2. Details
		12.2.3. Example | Impact of caffeine on pain
	12.3. Choosing a statistical model
		12.3.1. Mistake
		12.3.2. Details
		12.3.3. Mixed-effects
	12.4. Mistakes in estimating τ2
		12.4.1. Mistake
		12.4.2. Details
	12.5. Comparing the effect size in different subgroups
		12.5.1. Mistake
		12.5.2. Details
		12.5.3. Weight loss
		12.5.4. Caffeine
	12.6. Reporting an overall effect size in the presence of subgroups
		12.6.1. Mistake
		12.6.2. Details
	12.7. Putting it all together
Summary
Summary
Summary
Summary
13. Comprehensive Meta-Analysis Software
	13.1. Introduction
	13.2. Features in CMA
	13.3. Teaching elements
	13.4. Documentation
	13.5. Availability
	13.6. Acknowledgements
	13.7. Motivating example
	13.8. Data entry
	13.9. Basic analysis
		13.9.1. What is the average effect size?
		13.9.2. How much does the effect size vary?
	13.10. High-resolution plot
	13.11. Subgroup analysis
	13.12. Meta-regression
	13.13. Publication bias
14. How to report the results of an analysis
	14.1. Introduction
		14.1.1. The overview
		14.1.2. The mean effect size
		14.1.3. Variation in effect size
		14.1.4. Differences among subgroups
		14.1.5. Publication bias
		14.1.6. References
	14.2. Methylphenidate for adults with ADHD
	14.3. High dose vs. standard dose of statins
	14.4. Intervention to prevent alcohol abuse
	14.5. Prevalence of PTSD
	14.6. Training children to avoid sexual abuse
	14.7. Impact of interventions on truancy
	14.8. Correlation between commitment and performance
	14.9. Viagra for erectile dysfunction
	14.10. Second-hand smoking and lung cancer
	14.11. Weight loss by drug
15. Glossary
16. Appendix
	16.1. Appendix I – How statistical model affects confidence interval
		16.1.1. Random-effects vs. fixed-effects (plural)
	16.2. Appendix II –  Standard error of the summary effect
	16.3. Appendix III – How statistical model affects estimate of mean
	16.4. Appendix IV − Risk of bias
	16.5. Appendix V – Knapp-Hartung Adjustment
	16.6. Appendix VI – Statistics for heterogeneity
	16.7. Appendix VII – Computing a prediction interval
	16.8. Appendix VIII − Computing I2
	16.9. Appendix IX – Pairwise comparisons
		16.9.1. When the effect size is analyzed in the original metric
		16.9.2. When the effect size is a ratio
		16.9.3. Correlations
		16.9.4. Prevalence
		16.9.5. Multiple tests
	16.10. Appendix X – Pooling estimates of T2
		16.10.1. Combining subgroups
17. References
18. Subject index
19. Author index