Python for Natural Language Processing: Programming with NumPy, scikit-learn, Keras, and PyTorch (Cognitive Technologies)

Preface to the Third Edition
Preface to the Second Edition
Preface to the First Edition
Contents
1 An Overview of Language Processing
	1.1 Applications of Language Processing
	1.2 Evaluating the Applications
	1.3 Why Speech and Language Processing Are Difficult
		1.3.1 Ambiguity
		1.3.2 Models and Their Implementation
	1.4 The Domains We Will Cover
	1.5 Further Reading
2 A Tour of Python
	2.1 Why Python?
	2.2 The Read, Evaluate, and Print Loop
	2.3 Introductory Programs
	2.4 Strings
		2.4.1 String Index
		2.4.2 String Operations and Functions
		2.4.3 Slices
		2.4.4 Special Characters
		2.4.5 Formatting Strings
	2.5 Data Identities and Types
	2.6 Data Structures
		2.6.1 Lists
		2.6.2 List Copy
		2.6.3 Built-in List Operations and Functions
		2.6.4 Tuples
		2.6.5 Sets
		2.6.6 Built-in Set Functions
		2.6.7 Dictionaries
		2.6.8 Built-in Dictionary Functions
		2.6.9 Counting the Letters of a Text
	2.7 Control Structures
		2.7.1 Conditionals
		2.7.2 The for Loop
		2.7.3 The while Loop
		2.7.4 Exceptions
	2.8 Functions
	2.9 Documenting Functions
		2.9.1 Docstrings
		2.9.2 Type Annotations
	2.10 Comprehensions and Generators
		2.10.1 Comprehensions
		2.10.2 Generators
		2.10.3 Iterators
		2.10.4 zip
	2.11 Modules
	2.12 Installing Modules
	2.13 Basic File Input/Output
	2.14 Collecting a Corpus from the Internet
	2.15 Memo Functions and Decorators
		2.15.1 Memo Functions
		2.15.2 Decorators
	2.16 Object-Oriented Programming
		2.16.1 Classes and Objects
		2.16.2 Subclassing
		2.16.3 Counting with the Counter Class
	2.17 Functional Programming
		2.17.1 map()
		2.17.2 Lambda Expressions
		2.17.3 reduce()
		2.17.4 filter()
	2.18 Further Reading
3 Corpus Processing Tools
	3.1 Corpora
		3.1.1 Types of Corpora
		3.1.2 Corpora and Lexicon Building
		3.1.3 Corpora as Knowledge Sources
	3.2 Finite-State Automata
		3.2.1 A Description
		3.2.2 Mathematical Definition of Finite-State Automata
		3.2.3 Deterministic and Nondeterministic Automata
		3.2.4 Building a Deterministic Automaton from a Nondeterministic One
		3.2.5 Searching a String with a Finite-State Automaton
		3.2.6 Operations on Finite-State Automata
	3.3 Regular Expressions
		3.3.1 Repetition Metacharacters
		3.3.2 The Dot Metacharacter
		3.3.3 The Escape Character
		3.3.4 The Longest Match
		3.3.5 Character Classes
			Negated Character Classes
			Range of Characters
			Metacharacters
			Predefined Character Classes
		3.3.6 Nonprintable Symbols or Positions
		3.3.7 Union and Boolean Operators
		3.3.8 Operator Combination and Precedence
	3.4 Programming with Regular Expressions
		3.4.1 Matching
			The m/pattern/ Operator
			The m/pattern/g Operator
		3.4.2 A Simplified grep Program
		3.4.3 Match Modifiers
		3.4.4 Substitutions
		3.4.5 Backreferences
		3.4.6 Backreferences in the Pattern
		3.4.7 Raw Strings
		3.4.8 Python Escape Sequences
		3.4.9 Backreferences and Substitutions
		3.4.10 Match Objects
		3.4.11 Parameterable Regular Expressions
	3.5 Finding Concordances
	3.6 Lookahead and Lookbehind
	3.7 Approximate String Matching
		3.7.1 Edit Operations
		3.7.2 Edit Operations for Spell Checking
		3.7.3 Minimum Edit Distance
		3.7.4 Computing the Minimum Edit Distance in Python
		3.7.5 Searching Edits
	3.8 Further Reading
4 Encoding and Annotation Schemes
	4.1 Character Sets
		4.1.1 Representing Characters
		4.1.2 Unicode
		4.1.3 Character Composition and Normalization
		4.1.4 Unicode Character Properties
		4.1.5 The Unicode Encoding Schemes
	4.2 Locales and Word Order
		4.2.1 Presenting Time, Numerical Information, and Ordered Words
		4.2.2 The Unicode Collation Algorithm
		4.2.3 Sorting with Python
	4.3 Tabular Formats
		4.3.1 The csv Module
		4.3.2 Pandas
	4.4 Markup Languages
		4.4.1 An Outline of XML
			Elements
			Attributes
			Entities
		4.4.2 XML and Databases
	4.5 Collecting Corpora from the Web
		4.5.1 Scraping Documents with Python
		4.5.2 HTML
		4.5.3 Parsing HTML
	4.6 Further Reading
5 Python for Numerical Computations
	5.1 Dataset
	5.2 Vectors
		5.2.1 Representing the Counts
		5.2.2 Data Types
		5.2.3 Size of the Vectors
		5.2.4 Indices and Slices
		5.2.5 Operations
		5.2.6 Comparison with Lists
		5.2.7 PyTorch
			PyTorch Tensors
			PyTorch Data Types
			Size in PyTorch
			Indices in PyTorch
			NumPy/PyTorch Conversion
			PyTorch Device
		5.2.8 Mathematical Background: The Vector Space
	5.3 NumPy Functions
	5.4 The Dot Product
		5.4.1 Euclidian Norm of a Vector
		5.4.2 Cosine of Two Vectors
		5.4.3 The Dot Product in Mathematics
		5.4.4 Elementwise Product
	5.5 Matrices
		5.5.1 Matrices in NumPy
		5.5.2 Indices and Slices
		5.5.3 Order and Dimensions of a Tensor
		5.5.4 Addition and Multiplication by a Scalar
		5.5.5 Matrices in PyTorch
		5.5.6 Matrix Creation Functions
		5.5.7 Applying Functions
		5.5.8 Transposing and Reshaping Arrays
		5.5.9 Reshaping with PyTorch
		5.5.10 Broadcasting
	5.6 Matrix Products
		5.6.1 Matrix-Vector Multiplication
		5.6.2 Matrix Multiplication
		5.6.3 Computing the Cosines
	5.7 Elementary Mathematical Background for Matrices
		5.7.1 Linear and Affine Maps
		5.7.2 Linear Functions and Vectors
		5.7.3 Matrix Example
		5.7.4 Transpose
		5.7.5 Matrices and Rotations
		5.7.6 Function Composition
		5.7.7 Application to Rotations
		5.7.8 Inverse Function
		5.7.9 Inverting a Matrix in NumPy and PyTorch
	5.8 Application to Neural Networks
		5.8.1 Matrices and Datasets
		5.8.2 Matrices and PyTorch: One Layer
		5.8.3 More Layers
	5.9 Automatic Differentiation
	5.10 Further Reading
6 Topics in Information Theory and Machine Learning
	6.1 Codes and Information Theory
		6.1.1 Entropy
		6.1.2 Python Implementation
		6.1.3 Huffman Coding
		6.1.4 Cross-Entropy
		6.1.5 Perplexity and Cross-Perplexity
	6.2 Entropy and Decision Trees
		6.2.1 A Toy Dataset
		6.2.2 Decision Trees
		6.2.3 Inducing Decision Trees Automatically
		6.2.4 Numerical Attributes
	6.3 Encoding Categorical Values as Numerical Features
	6.4 Programming: Inducing Decision Trees with Scikit-Learn
		6.4.1 Conversion of Categorical Data
		6.4.2 Inducing a Decision Tree with Scikit-Learn
		6.4.3 Evaluating a Model
	6.5 Further Reading
7 Linear and Logistic Regression
	7.1 Linear Classifiers
	7.2 Choosing a Dataset
	7.3 Linear Regression
		7.3.1 Least Squares
		7.3.2 Least Absolute Deviation
	7.4 Notations in an n-Dimensional Space
	7.5 Gradient Descent
		7.5.1 Mathematical Description
		7.5.2 Gradient Descent and Linear Regression
			In a Two-Dimensional Space
			N-Dimensional Space
	7.6 Regularization
		7.6.1 The Analytical Solution Again
		7.6.2 Inverting XX
		7.6.3 Regularization
	7.7 Linear Classification
		7.7.1 An Example
		7.7.2 Classification in an N-Dimensional Space
		7.7.3 Linear Separability
		7.7.4 Classification vs. Regression
	7.8 Perceptron
		7.8.1 The Heaviside Function
		7.8.2 The Iteration
		7.8.3 The Two-Dimensional Case
		7.8.4 Stop Conditions
	7.9 Logistic Regression
		7.9.1 Fitting the Weight Vector
			Maximizing the Likelihood
		7.9.2 Gradient Ascent …
			…and Descent
			Computing the Gradient
			Weight Updates
	7.10 Gradient Descent Optimization
		7.10.1 The Momentum
		7.10.2 RMSprop
	7.11 Programming Logistic Regression with Scikit-Learn
		7.11.1 Representing the Dataset
		7.11.2 Scaling the Data
		7.11.3 Loading the Dataset from a File
			Loading a TSV File with Pandas
			The Svmlight Format
		7.11.4 Fitting a Model with Scikit-Learn
		7.11.5 The Logistic Regression Model
		7.11.6 The Loss
		7.11.7 Comparing Cross Entropy and the Squared Error
		7.11.8 Multinomial Logistic Regression
	7.12 Evaluation of Classification Systems
		7.12.1 Accuracy
		7.12.2 Precision and Recall
	7.13 Further Reading
8 Neural Networks
	8.1 Representation and Notation
	8.2 Feed-Forward Computation
		8.2.1 The Perceptron and Logistic Regression
		8.2.2 Hidden Layers
	8.3 Backpropagation
		8.3.1 Presentation
		8.3.2 Naive Gradient Descent
		8.3.3 Breaking Down the Computation
		8.3.4 Gradient with Respect to the Input
		8.3.5 Gradient with Respect to the Weights
	8.4 Applying Neural Networks to Datasets
	8.5 Programming Neural Networks
		8.5.1 Data Representation and Preprocessing
		8.5.2 Keras
			Building the Network
			Setting the Gradient Descent Parameters
			Training the Model
			Predicting Classes
			Model Parameters
			Adding Hidden Layers
		8.5.3 PyTorch
			Building the Network with Sequential
			Setting the Gradient Descent Parameters
			Compiling the Model
			Training the Model
			Predicting Classes
			Model Parameters
			Adding Hidden Layers
			PyTorch Dataloaders
			Building the Network: Deriving the Module Class
			Adding Hidden Layers
	8.6 Classification with More than Two Classes
		8.6.1 Cross Entropy Loss
		8.6.2 The Softmax Function
	8.7 Multiclass Classification with Keras
	8.8 Multiclass Classification with PyTorch
	8.9 Backpropagation in PyTorch
	8.10 Further Reading
9 Counting and Indexing Words
	9.1 Text Segmentation
		9.1.1 What Is a Word?
		9.1.2 Breaking a Text into Words and Sentences
	9.2 Tokenizing Words
		9.2.1 Defining Content
		9.2.2 Using Boundaries
		9.2.3 Improving Tokenization
		9.2.4 Tokenizing Using Classifiers
	9.3 Sentence Segmentation
		9.3.1 The Ambiguity of the Period Sign
		9.3.2 Rules To Disambiguate the Period Sign
		9.3.3 Using Regular Expressions
		9.3.4 Improving the Segmenter Using Lexicons
		9.3.5 Sentence Detection Using Classifiers
	9.4 Word Counting
		9.4.1 Some Definitions
		9.4.2 Counting Words with Python
		9.4.3 The Counter Class
		9.4.4 A Crash Program To Count Words with Unix
	9.5 Retrieval and Ranking of Documents
		9.5.1 Document Indexing
		9.5.2 Building an Inverted Index in Python
		9.5.3 Representing Documents as Vectors
		9.5.4 Vector Coordinates
		9.5.5 Ranking Documents
	9.6 Categorizing Text
		9.6.1 Corpora
		9.6.2 Building a Categorizer with Scikit-Learn
	9.7 Further Reading
10 Word Sequences
	10.1 Modeling Word Sequences
	10.2 N-Grams
		10.2.1 Counting Bigrams with Python
		10.2.2 Counting Bigrams with Unix
	10.3 Probabilistic Models of a Word Sequence
		10.3.1 The Maximum Likelihood Estimation
		10.3.2 Using ML Estimates with Nineteen Eighty-Four
			Training and Testing the Language Model
			Marking up the Corpus
			The Vocabulary
			Computing a Sentence Probability
	10.4 Smoothing N-Gram Probabilities
		10.4.1 Sparse Data
		10.4.2 Laplace\'s Rule
		10.4.3 Good–Turing Estimation
	10.5 Using N-Grams of Variable Length
		10.5.1 Linear Interpolation
		10.5.2 Back-Off
		10.5.3 Katz\'s Back-Off Model
		10.5.4 Kneser–Ney Smoothing Model
	10.6 Industrial N-Grams
	10.7 Quality of a Language Model
		10.7.1 Intuitive Presentation
		10.7.2 Entropy Rate
		10.7.3 Cross Entropy
		10.7.4 Perplexity
	10.8 Generating Text from a Language Model
		10.8.1 Using the Multinomial Distribution
		10.8.2 Transforming the Distribution
	10.9 Collocations
		10.9.1 Word Preference Measurements
			Mutual Information
			t-Scores
			Likelihood Ratio
		10.9.2 Extracting Collocations with Python
			Mutual Information
			t-Scores
			Log Likelihood Ratio
		10.9.3 Applying Collocation Measures
	10.10 Further Reading
11 Dense Vector Representations
	11.1 Vector Representations
	11.2 Dimensionality Reduction
		11.2.1 Singular Value Decomposition
		11.2.2 Data Representation and Preprocessing
		11.2.3 Computing a Singular Value Decomposition
	11.3 Applying a SVD to the Salammbô Dataset
		11.3.1 Counts of Letter A
		11.3.2 Counts of all the Characters
		11.3.3 The Characters in a Space of Documents
		11.3.4 Singular Value Decomposition and Principal Component Analysis
	11.4 Latent Semantic Indexing
	11.5 Word Embeddings from a Cooccurrence Matrix
		11.5.1 Preprocessing
		11.5.2 Counting the Cooccurrences
		11.5.3 Applying a PCA
		11.5.4 Saving the Vectors
	11.6 Embeddings\' Similarity
		11.6.1 Cosine Similarity
		11.6.2 Programming
	11.7 From Cooccurrences to Mutual Information
	11.8 GloVe
		11.8.1 Model
		11.8.2 Loss
		11.8.3 Embeddings
		11.8.4 The Dataloader
		11.8.5 Programming the Model
		11.8.6 Computing the Embeddings
		11.8.7 Semantic Similarity
	11.9 Word Embeddings from Neural Networks
		11.9.1 word2vec
		11.9.2 CBOW Architecture
		11.9.3 Programming CBOW
			Preprocessing the Text
			Embeddings
			Embedding Bags
			The Network
			CBOW Embeddings
			Semantic Similarity
		11.9.4 Skipgrams
			The Model
			Skipgrams Negative Sampling
			The Network
			The Loss
			Dataset Preparation
			Semantic Similarity
	11.10 Application of Embeddings to Language Detection
		11.10.1 Corpus
		11.10.2 Preprocessing and Balancing the Dataset
		11.10.3 Character N-Grams
		11.10.4 Encoding the N-Grams
		11.10.5 Building X and y
		11.10.6 Bags of Embeddings
		11.10.7 Model
		11.10.8 Training Loop
		11.10.9 Evaluation
	11.11 Further Reading
12 Words, Parts of Speech, and Morphology
	12.1 Words
		12.1.1 Parts of Speech
		12.1.2 Grammatical Features
		12.1.3 Two Significant Parts of Speech: The Noun and the Verb
			The Noun
			Verbs
	12.2 Standardized Part-of-Speech Tagsets and Grammatical Features
		12.2.1 Multilingual Part-of-Speech Tags
		12.2.2 Multilingual Grammatical Features
	12.3 The CoNLL Format
	12.4 A CoNLL Reader in Python
	12.5 Lexicons
		12.5.1 Encoding a Dictionary
	12.6 Morphology
		12.6.1 Morphemes
		12.6.2 Morphs
		12.6.3 Inflection and Derivation
			Some Definitions
			Inflection
			Derivation
			Morphological Processing
			Ambiguity
		12.6.4 Language Differences
	12.7 Morphological Parsing
		12.7.1 Two-Level Model of Morphology
		12.7.2 Interpreting the Morphs
		12.7.3 Finite-State Transducers
		12.7.4 Conjugating a French Verb
		12.7.5 Application to Romance Languages
		12.7.6 Ambiguity
		12.7.7 Operations on Finite-State Transducers
	12.8 Further Reading
13 Subword Segmentation
	13.1 Deriving Morphemes Automatically
	13.2 Byte-Pair Encoding
		13.2.1 Outline of the Algorithm
		13.2.2 Pretokenization
		13.2.3 The Initial Vocabulary
		13.2.4 Counting the Bigrams
		13.2.5 Merging Pairs
		13.2.6 Constructing the Merge Rules
		13.2.7 Encoding
		13.2.8 Tokenizing
		13.2.9 Visualizing the Whitespaces
		13.2.10 Using Bytes
	13.3 The WordPiece Tokenizer
		13.3.1 Pretokenization and Initial Vocabulary
		13.3.2 Computing the Gains
		13.3.3 Constructing the Subwords
		13.3.4 Encoding
		13.3.5 Tokenization
		13.3.6 BERT\'s WordPiece
	13.4 Unigram Tokenizer
		13.4.1 Initial Class
		13.4.2 Estimating the Probabilities
		13.4.3 Finding the Best Segmentation
			Brute-Force Search
			Viterbi Algorithm
		13.4.4 Expectation-Maximization (EM)
		13.4.5 Creating the Vocabulary
	13.5 The SentencePiece Tokenizer
	13.6 Hugging Face Tokenizers
		13.6.1 Pretrained BPE
		13.6.2 Training BPE
	13.7 Further Reading
14 Part-of-Speech and Sequence Annotation
	14.1 Resolving Part-of-Speech Ambiguity
	14.2 Baseline
	14.3 Evaluation
	14.4 Part-of-Speech Tagging with Linear Classifiers
	14.5 Programming a Part-of-Speech Taggerwith Logistic Regression
		14.5.1 Building the Feature Vectors
		14.5.2 Encoding the Features
		14.5.3 Applying the Classifier
	14.6 Part-of-Speech Tagging with Feed-Forward Networks
		14.6.1 Programming a Single Layer Network for POS Tagging
		14.6.2 Training the Model
		14.6.3 Networks with Hidden Layers
	14.7 Embeddings
	14.8 Recurrent Neural Networks
		14.8.1 Programming RNNs for POS Tagging
			Indices
			Padding
			Embeddings
			Network Architecture
			Training Loop
		14.8.2 Dropout
		14.8.3 LSTM
		14.8.4 POS Tagging with LSTMs
		14.8.5 Further Improvements
	14.9 Named Entity Recognition and Chunking
		14.9.1 Noun Groups and Verb Groups
		14.9.2 Named Entities
	14.10 Group Annotation Using Tags
		14.10.1 The IOB Tagset
		14.10.2 The IOB2 Tagset
		14.10.3 The BIOES Tagset
		14.10.4 Extending BIO to Two or More Groups
		14.10.5 Annotation Examples from CoNLL 2000, 2002, and 2003
	14.11 Recurrent Networks for Named Entity Recognition
	14.12 Conditional Random Fields
		14.12.1 Modeling the y Transitions
		14.12.2 Adding the x Input
		14.12.3 Predicting a Sequence
		14.12.4 Adding a CRF Layer to a LSTM Network
	14.13 Tokenization
	14.14 Further Reading
15 Self-Attention and Transformers
	15.1 The Transformer Architecture
	15.2 Self-Attention
		15.2.1 GloVe Static Embeddings
		15.2.2 Using Weighted Cosines to Create Contextual Embeddings
		15.2.3 Dot-Product Attention
		15.2.4 Projecting the Input
		15.2.5 Programming the Attention Function
		15.2.6 Adding the Query, Key, and Value Matrices
	15.3 Multihead Attention
	15.4 Residual Connections
	15.5 Feedforward Sublayer
	15.6 The Encoder Layers
	15.7 The Complete Encoder
	15.8 Input Embeddings
	15.9 Positional Encodings
	15.10 Converting the Input Words
	15.11 Improving the Encoder
	15.12 PyTorch Transformer Modules
		15.12.1 The PyTorch Multihead Attention Class
		15.12.2 The PyTorch TransformerEncoder Class
	15.13 Application: Sequence Annotation
	15.14 Application: Classification
	15.15 Further Reading
16 Pretraining an Encoder: The BERT Language Model
	16.1 Pretraining Tasks
	16.2 Creating the Dataset
		16.2.1 Input Sequence
		16.2.2 Next Sentence Prediction
		16.2.3 Masked Language Model
	16.3 The Input Embeddings
	16.4 Creating a Batch
	16.5 BERT Architecture
	16.6 BERT Pretraining
	16.7 The Training Loop
	16.8 BERT Applications
		16.8.1 Classification
		16.8.2 Sequence Annotation
		16.8.3 Question Answering
		16.8.4 Question Answering Systems
	16.9 Using a Pretrained Model
	16.10 The IMDB Dataset
	16.11 Tokenization
	16.12 Fine-Tuning
		16.12.1 Architecture
		16.12.2 Training Parameters
		16.12.3 Training
		16.12.4 Prediction
		16.12.5 Freezing Layers
	16.13 Further Reading
17 Sequence-to-Sequence Architectures: Encoder-Decoders and Decoders
	17.1 Parallel Corpora
	17.2 Alignment
	17.3 The Encoder-Decoder Architecture
	17.4 Encoder-Decoder Transformers
		17.4.1 Masked Attention
		17.4.2 Cross-Attention
		17.4.3 Evaluating Translation
	17.5 Programming: Machine Translation
		17.5.1 The PyTorch TransformerDecoder Class
		17.5.2 The PyTorch Transformer Class
		17.5.3 Preparing the Dataset
		17.5.4 Indexing the Symbols
		17.5.5 Building the Transformer Model
		17.5.6 Configuring the Model and Training Procedure
		17.5.7 Datasets and Dataloaders
		17.5.8 Training the Model
		17.5.9 Decoding a Sentence
		17.5.10 Improving the Performance
	17.6 Decoders
		17.6.1 Decoder Architecture
		17.6.2 Training Procedure
	17.7 Further Developments
References
Index