Statistics in Corpus Linguistics Research: A New Approach

Cover
Half Title
Title Page
Copyright Page
Table of contents
Preface
Acknowledgments
A Note on Terminology and Notation
Part 1 Motivations
	1 What Might Corpora Tell Us About Language?
		1.1 Introduction
		1.2 What Might a Corpus Tell Us?
		1.3 The 3A Cycle
			1.3.1. Annotation, Abstraction and Analysis
			1.3.2 The Problem of Representational Plurality
			1.3.3 ICECUP: A Platform for Treebank Research
		1.4 What Might a Richly Annotated Corpus Tell Us?
		1.5 External Influences: Modal Shall / Will Over Time
		1.6 Interacting Grammatical Decisions: NP Premodification
		1.7 Framing Constraints and Interaction Evidence
			1.7.1 Framing Frequency Evidence
			1.7.2 Framing Interaction Evidence
			1.7.3 Framing and Annotation
			1.7.4 Framing and Sampling
		1.8 Conclusions
		Notes
Part 2 Designing Experiments with Corpora
	2 The Idea of Corpus Experiments
		2.1 Introduction
		2.2 Experimentation and Observation
			2.2.1 Obtaining Data
			2.2.2 Research Questions and Hypotheses
			2.2.3 From Hypothesis to Experiment
		2.3 Evaluating a Hypothesis
			2.3.1 The Chi-Square Test
			2.3.2 Extracting Data
			2.3.3 Visualising Proportions, Probabilities and Significance
		2.4 Refining the Experiment
		2.5 Correlations and Causes
		2.6 A Linguistic Interaction Experiment
		2.7 Experiments and Disproof
		2.8 What Is the Point of an Experiment?
		2.9 Conclusions
		Notes
	3 That Vexed Problem of Choice
		3.1 Introduction
			3.1.1 The Traditional ‘Per Million Words’ Approach
			3.1.2 How Did Per Million Word Statistics Become Dominant?
			3.1.3 Choice Models and Linguistic Theory
			3.1.4 The Vexed Problem of Choice
			3.1.5 Exposure Rates and Other Experimental Models
			3.1.6 What Do We Mean by ‘Choice’?
		3.2 Parameters of Choice
			3.2.1 Types of Mutual Substitution
			3.2.2 Multi-Way Choices and Decision Trees
			3.2.3 Binomial Statistics, Tests and Time Series
			3.2.4 Lavandera’s Dangerous Hypothesis
		3.3 A Methodological Progression?
			3.3.1 Per Million Words
			3.3.2 Selecting a More Plausible Baseline
			3.3.3 Enumerating Alternates
			3.3.4 Linguistically Restricting the Sample
			3.3.5 Eliminating Non-Alternating Cases
			3.3.6 A Methodological Progression
		3.4 Objections to Variationism
			3.4.1 Feasibility
			3.4.2 Arbitrariness
			3.4.3 Oversimplification
			3.4.4 The Problem of Polysemy
			3.4.5 A Complex Ecology?
			3.4.6 Necessary Reductionism Versus Complex Statistical Models
			3.4.7 Discussion
		3.5 Conclusions
		Notes
	4 Choice Versus Meaning
		4.1 Introduction
		4.2 The Meanings of Very
		4.3 The Choices of Very
		4.4 Refining Baselines by Type
		4.5 Conclusions
	5 Balanced Samples and    Imagined Populations
		5.1 Introduction
		5.2 A Study in Genre Variation
		5.3 Imagining Populations
		5.4 Multi-Variate and Multi-Level Modelling
		5.5 More Texts – or Longer Ones?
		5.6 Conclusions
Part 3 Confidence Intervals and Significance Tests
	6 Introducing Inferential  Statistics
		6.1 Why Is Statistics Difficult?
		6.2 The Idea of Inferential Statistics
		6.3 The Randomness of Life
			6.3.1 The Binomial Distribution
			6.3.2 The Ideal Binomial Distribution
			6.3.3 Skewed Distributions
			6.3.4 From Binomial to Normal
			6.3.5 From Gauss to Wilson
			6.3.6 Scatter and Confidence
		6.4 Conclusions
		Notes
	7 Plotting With Confidence
		7.1 Introduction
			7.1.1 Visualising Data
			7.1.2 Comparing Observations and Identifying Significant Differences
		7.2 Plotting the Graph
			7.2.1 Step 1. Gather Raw Data
			7.2.2 Step 2. Calculate Basic Wilson Score Interval Terms
			7.2.3 Step 3. Calculate the Wilson Interval
			7.2.4 Step 4. Plotting Intervals on Graphs
		7.3 Comparing and Plotting Change
			7.3.1 The Newcombe-Wilson Interval
			7.3.2 Comparing Intervals: An Illustration
			7.3.3 What Does the Newcombe-Wilson Interval Represent?
			7.3.4 Comparing Multiple Points
			7.3.5 Plotting Percentage Difference
			7.3.6 Floating Bar Charts
		7.4 An Apparent Paradox
		7.5 Conclusions
		Notes
	8 From Intervals to Tests
		8.1 Introduction
			8.1.1 Binomial Intervals and Tests
			8.1.2 Sampling Assumptions
				Assumption 1. Randomness and independence
				Assumption 2. Every sampled instance is free to vary
				Assumption 3. The sample is very small relative to the size of the population
			8.1.3 Deriving a Binomial Distribution
			8.1.4 Some Example Data
		8.2 Tests for a Single Binomial Proportion
			8.2.1 The Single-Sample z Test
			8.2.2 The 2 × 1 Goodness of Fit .2 Test
			8.2.3 The Wilson Score Interval
			8.2.4 Correcting for Continuity
			8.2.5 The ‘Exact’ Binomial Test
			8.2.6 The Clopper-Pearson Interval
			8.2.7 The Log-Likelihood Test
			8.2.8 A Simple Performance Comparison
		8.3 Tests for Comparing Two Observed Proportions
			8.3.1 The 2 × 2 .2 and z Test for Two Independent Proportions
			8.3.2 The z Test for Two Independent Proportions from Independent       Populations
			8.3.3 The z Test for Two Independent Proportions with a Given Difference in Population Means
			8.3.4 Continuity-Corrected 2 × 2 Tests
			8.3.5 The Fisher ‘Exact’ Test
		8.4 Applying Contingency Tests
			8.4.1 Selecting Tests
			8.4.2 Analysing Larger Tables
			8.4.3 Linguistic Choice
			8.4.4 Case Interaction
			8.4.5 Large Samples and Small Populations
		8.5 Comparing the Results of Experiments
		8.6 Conclusions
		Notes
	9 Comparing Frequencies in the Same Distribution
		9.1 Introduction
		9.2 The Single-Sample z Test
			9.2.1 Comparing Frequency Pairs for Significant Difference
			9.2.2 Performing the Test
		9.3 Testing and Interpreting Intervals
			9.3.1 The Wilson Interval Comparison Heuristic
			9.3.2 Visualising the Test
		9.4 Conclusions
		Notes
	10 Reciprocating the  Wilson Interval
		10.1 Introduction
		10.2 The Wilson Interval of Mean Utterance Length
			10.2.1 Scatter and Confidence
			10.2.2 From Length to Proportion
			10.2.3 Example: Confidence Intervals on Mean Length of Utterance
			10.2.4 Plotting the Results
		10.3 Intervals on Monotonic Functions of p
		10.4 Conclusions
		Notes
	11 Competition Between    Choices Over Time
		11.1 Introduction
		11.2 The ‘S Curve’
		11.3 Boundaries and Confidence Intervals
			11.3.1 Confidence Intervals for p
			11.3.2 Logistic Curves and Wilson Intervals
		11.4 Logistic Regression
			11.4.1 From Linear to Logistic Regression
			11.4.2 Logit-Wilson Regression
			11.4.3 Example 1: The Decline of the To-infinitive Perfect
			11.4.4 Example 2: Catenative Verbs in Competition
			11.4.5 Review
		11.5 Impossible Logistic Multinomials
			11.5.1 Binomials
			11.5.2 Impossible Multinomials
			11.5.3 Possible Hierarchical Multinomials
			11.5.4 A Hierarchical Reanalysis of Example 2
			11.5.5 The Three-Body Problem
		11.6 Conclusions
		Notes
	12 The Replication Crisis and      the New Statistics
		12.1 Introduction
		12.2 A Corpus Linguistics Debate
		12.3 Psychology Lessons?
		12.4 The Road Not Travelled
		12.5 What Does This Mean for Corpus Linguistics?
		12.6 Some Recommendations
			12.6.1 Recommendation 1: Include a Replication Step
			12.6.2 Recommendation 2: Focus on Large Effects – and Clear Visualisations
			12.6.3 Recommendation 3: Play Devil’s Advocate
			12.6.4 A Checklist for Empirical Linguistics
		12.7 Conclusions
		Notes
	13 Choosing the Right Test
		13.1 Introduction
			13.1.1 Choosing a Dependent Variable and Baseline
			13.1.2 Choosing Independent Variables
		13.2 Tests for Categorical Data
			13.2.1 Two Types of Contingency Test
			13.2.2 The Benefits of Simple Tests
			13.2.3 Visualising Uncertainty
			13.2.4 When to Use Goodness of Fit Tests
			13.2.5 Tests for Comparing Results
			13.2.6 Optimum Methods of Calculation
		13.3 Tests for Other Types of Data
			13.3.1 t Tests for Comparing Two Independent Samples of Numeric Data
			13.3.2 Reversing Tests
			13.3.3 Tests for Other Types of Variables
			13.3.4 Quantisation
		13.4 Conclusions
		Notes
Part 4 Effect Sizes and Meta-Tests
	14 The Size of an Effect
		14.1 Introduction
		14.2 Effect Sizes for Two-Variable Tables
			14.2.1 Simple Difference
			14.2.2 The Problem of Prediction
			14.2.3 Cramér’s .
			14.2.4 Other Probabilistic Approaches to Dependent Probability
		14.3 Confidence Intervals on .
			14.3.1 Confidence Intervals on 2 × 2 .
			14.3.2 Confidence Intervals for Cramér’s .
			14.3.3 Example: Investigating Grammatical Priming
		14.4 Goodness of Fit Effect Sizes
			14.4.1 Unweighted .p
			14.4.2 Variance-Weighted .e
			14.4.3 Example: Correlating the Present Perfect
		14.5 Conclusions
		Notes
	15 Meta-Tests for Comparing    Tables of Results
		15.1 Introduction
			15.1.1 How Not to Compare Test Results
			15.1.2 Comparing Sizes of Effect
			15.1.3 Other Meta-Tests
		15.2 Some Preliminaries
			15.2.1 Test Assumptions
			15.2.2 Correcting for Continuity
			15.2.3 Example Data and Notation
		15.3 Point and Multi-Point Tests for Homogeneity Tables
			15.3.1 Reorganising Contingency Tables for 2 × 1 Tests
			15.3.2 The Newcombe-Wilson Point Test
			15.3.3 The Gaussian Point Test
			15.3.4 The Multi-Point Test for r × c Homogeneity Tables
		15.4 Gradient Tests for Homogeneity Tables
			15.4.1 The 2 × 2 Newcombe-Wilson Gradient Test
			15.4.2 Cramér’s . Interval and Test
			15.4.3 r × 2 Homogeneity Gradient Tests
			15.4.4 Interpreting Gradient Meta-Tests for Large Tables
		15.5 Gradient Tests for Goodness of Fit Tables
			15.5.1 The 2 × 1 Wilson Interval Gradient Test
			15.5.2 r × 1 Goodness of Fit Gradient Tests
		15.6 Subset Tests
			15.6.1 Point Tests for Subsets
			15.6.2 Multi-Point Subset Tests
			15.6.3 Gradient Subset Tests
			15.6.4 Goodness of Fit Subset Tests
		15.7 Conclusions
		Notes
Part 5 Statistical Solutions for 
Corpus Samples
	16 Conducting Research With Imperfect Data
		16.1 Introduction
		16.2 Reviewing Subsamples
			16.2.1 Example 1: Get Versus Be Passive
			16.2.2 Subsampling and Reviewing
			16.2.3 Estimating the Observed Probability p
			16.2.4 Contingency Tests and Multinomial Dependent Variables
		16.3 Reviewing Preliminary Analyses
			16.3.1 Example 2: Embedded and Sequential Postmodifiers
			16.3.2 Testing the Worst-Case Scenario
			16.3.3 Combining Subsampling Worst-Case Analysis
			16.3.4 Ambiguity and Error
		16.4 Resampling and p-hacking
		16.5 Conclusions
		Notes
	17 Adjusting Intervals for     Random-Text Samples
		17.1 Introduction
		17.2 Recalibrating Binomial Models
		17.3 Examples with Large Samples
			17.3.1 Example 1: Interrogative Clause Proportion, ‘Direct Conversations’
			17.3.2 Example 2: Clauses Per Word, ‘Direct Conversations’
			17.3.3 Uneven-Size Subsamples
			17.3.4 Example 1 Revisited, Across ICE-GB
		17.4 Alternation Studies with Small Samples
			17.4.1 Applying the Large Sample Method
			17.4.2 Singletons, Partitioning and Pooling
			17.4.3 Review
		17.5 Conclusions
		1601065483601_298
Part 6 Concluding Remarks
	18 Plotting the Wilson Distribution
		18.1 Introduction
		18.2 Plotting the Distribution
			18.2.1 Calculating w–(a) from the Standard Normal Distribution
			18.2.2 Plotting Points
			18.2.3 Delta Approximation
		18.3 Example Plots
			18.3.1 Sample Size n = 10, Observed Proportion p = 0.5
			18.3.2 Properties of Wilson Areas
			18.3.3 The Effect of p Tending to Extremes
			18.3.4 The Effect of Very Small n
		18.4 Further Perspectives on Wilson Distributions
			18.4.1 Percentiles of Wilson Distributions
			18.4.2 The Logit-Wilson Distribution
		18.5 Alternative Distributions
			18.5.1 Continuity-Corrected Wilson Distributions
			18.5.2 Clopper-Pearson Distributions
		18.6 Conclusions
		Notes
	19 In Conclusion
Appendices
1601065483601_321
1601065483601_322
Glossary
References
	Appendix AThe Interval Equality Principle
	Appendix BPseudo-Code for Computational Procedures
Index