Natural Language Processing in Action

Natural Language Processing in Action
Praise for the First Edition
brief contents
contents
preface
acknowledgments
about this book
	About the technology
	About this book
	Who should read this book
	How this book is organized: A road map
	Where to start
	About the code
	liveBook discussion forum
about the authors
about the contributors
about the cover illustration
Part 1 Wordy machines: Vector models of natural language
	1 Machines that read and write: A natural language processing overview
		1.1 Programming languages vs. NLP
			1.1.1 Natural language understanding
			1.1.2 Natural language generation
			1.1.3 Plumbing it all together for positive-impact AI
		1.2 The magic of natural language
			1.2.1 Language and thought
			1.2.2 Machines that converse
			1.2.3 The math
		1.3 Applications
			1.3.1 Processing programming languages with NLP
		1.4 Language through a computer’s “eyes”
			1.4.1 The language of locks
			1.4.2 Regular expressions
		1.5 Building a simple chatbot
			1.5.1 Keyword-based greeting recognizer
			1.5.2 Pattern-based intent recognition
			1.5.3 Another way to recognize greetings
		1.6 A brief overflight of hyperspace
		1.7 Word order and grammar
		1.8 A chatbot natural language pipeline
		1.9 Processing in depth
		1.10 Natural language IQ
		1.11 Test yourself
		Summary
	2 Tokens of thought: Natural language words
		2.1 Tokens and tokenization
			2.1.1 Your tokenizer toolbox
			2.1.2 The simplest tokenizer
			2.1.3 Rule-based tokenization
			2.1.4 SpaCy
			2.1.5 Finding the fastest word tokenizer
		2.2 Beyond word tokens
			2.2.1 WordPiece tokenizers
		2.3 Improving your vocabulary
			2.3.1 Extending your vocabulary with n-grams
			2.3.2 Normalizing your vocabulary
		2.4 Challenging tokens: Processing logographic languages
			2.4.1 A complicated picture: Lemmatization and stemming in Chinese
		2.5 Vectors of tokens
			2.5.1 One-hot vectors
			2.5.2 Bag-of-words vectors
			2.5.3 Why not bag of characters?
		2.6 Sentiment
			2.6.1 VADER: A rule-based sentiment analyzer
			2.6.2 Naive Bayes
		2.7 Test yourself
		Summary
	3 Math with words: Term frequency–inverse document frequency vectors
		3.1 Bag-of-words vectors
		3.2 Vectorizing text DataFrame constructor
			3.2.1 Faster, better, easier token counting
			3.2.2 Vectorizing your code
			3.2.3 Vector space TF–IDF (term frequency–inverse document frequency)
		3.3 Vector distance and similarity
			3.3.1 Dot product
		3.4 Counting TF–IDF frequencies
			3.4.1 Analyzing “this”
		3.5 Zipf’s law
		3.6 Inverse document frequency
			3.6.1 Return of Zipf
			3.6.2 Relevance ranking
			3.6.3 Smoothing out the math
		3.7 Using TF–IDF for your bot
		3.8 What’s next
		3.9 Test yourself
		Summary
	4 Finding meaning in word counts: Semantic analysis
		4.1 From word counts to topic scores
			4.1.1 The limitations of TF–IDF vectors and lemmatization
			4.1.2 Topic vectors
			4.1.3 Thought experiment
			4.1.4 Algorithms for scoring topics
		4.2 The challenge: Detecting toxicity
			4.2.1 Linear discriminant analysis classifier
			4.2.2 Going beyond linear
		4.3 Reducing dimensions
			4.3.1 Enter principal component analysis
			4.3.2 Singular value decomposition
		4.4 Latent semantic analysis
			4.4.1 Diving into semantics analysis
			4.4.2 TruncatedSVD or PCA?
			4.4.3 How well does LSA perform for toxicity detection?
			4.4.4 Other ways to reduce dimensions
		4.5 Latent Dirichlet allocation
			4.5.1 The LDiA idea
			4.5.2 LDiA topic model for comments
			4.5.3 Detecting toxicity with LDiA
			4.5.4 A fairer comparison: 32 LDiA topics
		4.6 Distance and similarity
		4.7 Steering with feedback
		4.8 Topic vector power
			4.8.1 Semantic search
		4.9 Equipping your bot with semantic search
		4.10 Test yourself
		Summary
Part 2 Deeper learning: Neural networks
	5 Word brain: Neural networks
		5.1 Why neural networks?
			5.1.1 Neural networks for words
			5.1.2 Neurons as feature engineers
			5.1.3 Biological neurons
			5.1.4 Perceptron
			5.1.5 A Python perceptron
		5.2 An example logistic neuron
			5.2.1 The logistics of clickbait
			5.2.2 Sex education
			5.2.3 Pronouns, gender, and sex
			5.2.4 Sex logistics
			5.2.5 A sleek, new PyTorch neuron
		5.3 Skiing down the error slope
			5.3.1 Off the chair lift, onto the slope: Gradient descent and local minima
			5.3.2 Shaking things up: Stochastic gradient descent
		5.4 Test yourself
		Summary
	6 Reasoning with word embeddings
		6.1 This is your brain on words
		6.2 Applications
			6.2.1 Search for meaning
			6.2.2 Combining word embeddings
			6.2.3 Analogy questions
			6.2.4 Word2Vec innovation
			6.2.5 Artificial intelligence relies on embeddings
		6.3 Word2Vec
			6.3.1 Analogy reasoning
			6.3.2 Learning word embeddings
			6.3.3 Learning meaning without a dictionary
			6.3.4 Using the gensim.word2vec module
			6.3.5 Generating your own word vector representations
		6.4 Word2Vec alternatives
			6.4.1 GloVe
			6.4.2 fastText
			6.4.3 Word2Vec vs. LSA
			6.4.4 Static vs. contextualized embeddings
			6.4.5 Visualizing word relationships
			6.4.6 Making connections
			6.4.7 Unnatural words
		6.5 Test yourself
		Summary
	7 Finding kernels of knowledge in text with CNNs
		7.1 Patterns in sequences of words
		7.2 Convolution
			7.2.1 Stencils for natural language text
			7.2.2 A bit more stenciling
			7.2.3 Correlation vs. convolution
			7.2.4 Convolution as a mapping function
			7.2.5 Python convolution example
			7.2.6 PyTorch 1D CNN on 4D embedding vectors
			7.2.7 Natural examples
		7.3 Morse code
			7.3.1 Decoding Morse with convolution
		7.4 Building a CNN with PyTorch
			7.4.1 Clipping and padding
			7.4.2 Better representation with word embeddings
			7.4.3 Transfer learning
			7.4.4 Robustifying your CNN with dropout
		7.5 PyTorch CNN to process disaster toots
			7.5.1 Network architecture
			7.5.2 Pooling
			7.5.3 Linear layers
			7.5.4 Getting fit
			7.5.5 Hyperparameter tuning
		7.6 Test yourself
		Summary
	8 Reduce, reuse, and recycle your words: RNNs and LSTMs
		8.1 What are RNNs good for?
			8.1.1 RNN sequence handling
			8.1.2 RNNs remember everything you tell them
			8.1.3 RNNs hide their understanding
			8.1.4 RNNs remember everything you tell them
		8.2 Predicting nationality with only a last name
			8.2.1 Building an RNN from scratch
			8.2.2 Training an RNN, one token at a time
			8.2.3 Understanding the results
			8.2.4 Multiclass classifiers vs. multi-label taggers
		8.3 Backpropagation through time
			8.3.1 Initializing the hidden layer in an RNN
		8.4 Remembering with recurrent networks
			8.4.1 Word-level language models
			8.4.2 Gated recurrent units
			8.4.3 Long short-term memory
			8.4.4 Giving your RNN a tune-up
		8.5 Predicting
		8.6 Test yourself
		Summary
Part 3 Getting real: Real-world NLP applications
	9 Stackable deep learning: Transformers
		9.1 Recursion vs. recurrence
			9.1.1 Attention is not all you need
			9.1.2 A LEGO set for language
		9.2 Filling the attention gaps
			9.2.1 Positional encoding
			9.2.2 Connecting all the pieces
			9.2.3 Transformer translation
		9.3 Bidirectional backpropagation and BERT
			9.3.1 Tokenization and pretraining
			9.3.2 Fine-tuning
			9.3.3 Implementation
			9.3.4 Fine-tuning a pretrained BERT model for text classification
		9.4 Test yourself
		Summary
	10 Large language models in the real world
		10.1 Large language models
			10.1.1 Scaling up
			10.1.2 Smarter, smaller LLMs
			10.1.3 Semantic routing and guard rails
			10.1.4 Red teaming
		10.2 Generating words with your own LLM
			10.2.1 Creating your own generative LLM
			10.2.2 Fine-tuning your generative model
			10.2.3 Nonsense: Hallucination
		10.3 Giving LLMs an IQ boost with search
			10.3.1 Searching for words: Full-text search
			10.3.2 Searching for meaning: Semantic search
			10.3.3 Scaling up your semantic search
			10.3.4 Approximate nearest neighbor search
			10.3.5 Choosing your index
			10.3.6 Quantizing the math
			10.3.7 Pulling it all together with Haystack
			10.3.8 Getting real
			10.3.9 A haystack of knowledge
			10.3.10 Answering questions
			10.3.11 Combining semantic search with text generation
			10.3.12 Deploying your app in the cloud
			10.3.13 Serve your users better
			10.3.14 AI ethics vs. AI safety
		10.4 Test yourself
		Summary
	11 Information extraction and knowledge graphs
		11.1 Grounding
			11.1.1 Going old-fashioned: Information extraction with patterns
		11.2 First things first: Segmenting your text into sentences
			11.2.1 Why won’t split(\'.!?\') work?
			11.2.2 Sentence segmentation with regular expressions
			11.2.3 Sentence semantics
		11.3 A knowledge extraction pipeline
		11.4 Entity recognition
			11.4.1 Pattern-based entity recognition: Extracting GPS locations
			11.4.2 Named entity recognition with spaCy
		11.5 Coreference resolution
			11.5.1 Coreference resolution with spaCy
			11.5.2 Entity name normalization
		11.6 Dependency parsing
			11.6.1 Constituency parsing with benepar
		11.7 From dependency parsing to relation extraction
			11.7.1 Pattern-based relation extraction
			11.7.2 Neural relation extraction
		11.8 Building your knowledge base
			11.8.1 A large knowledge graph
		11.9 Finding answers in a knowledge graph
			11.9.1 From questions to queries
		11.10 Test yourself
		Summary
	12 Getting chatty with dialog engines
		12.1 Chatbots are everywhere
			12.1.1 Different chatbots, same tools
			12.1.2 Conversation design
			12.1.3 Your first conversation diagram
			12.1.4 What makes a good conversation?
			12.1.5 Making your chatbot a good listener: Implicit and explicit confirmations
			12.1.6 Using GUI elements
		12.2 Making sense of the user’s input: Natural language understanding
			12.2.1 Intent recognition
			12.2.2 Multi-label classification
		12.3 Generating a response
			12.3.1 Template-based approach
			12.3.2 Conversation graphs
			12.3.3 Storing your graph in a relational database
			12.3.4 Scaling up the content: The search-based approach
			12.3.5 Designing more complex logic: The programmatic approach
		12.4 The generative approach
		12.5 Chatbot frameworks
			12.5.1 Building an intent-based chatbot with Rasa
			12.5.2 Adding LLMs to your chatbot with LangChain
		12.6 Maintaining your chatbot’s design
		12.7 Evaluating your chatbot
			12.7.1 Defining your chatbot’s performance metrics
			12.7.2 Measuring NLU performance
			12.7.3 Measuring user experience
			12.7.4 What’s next?
		12.8 Test yourself
		Summary
appendix A  Your NLP tools
	A.1 Installing nlpia2 in a virtual environment
	A.2 Batteries included
	A.3 Anaconda3
	A.4 Installing nlpia2
	A.5 An integrated development environment
	A.6 Debian package manager
	A.7 Macs
		A.7.1 A Mac package manager
		A.7.2 Some useful packages
	A.8 Tune-ups
		A.8.1 Ubuntu tune-ups
		A.8.2 Mac tune-ups
	A.9 Windows
		A.9.1 Chocolatey
		A.9.2 Get virtual
		A.9.3 Installing nlpia2 in a Linux container
appendix B Playful Python and regular expressions
	B.1 A playful development environment
		B.1.1 Start your REPL engines
		B.1.2 Exploring Python land
	B.2 Working with strings
		B.2.1 String types: str and bytes
		B.2.2 Templates in Python: .format()
	B.3 Mapping in Python: dict and OrderedDict
	B.4 Regular expression
		B.4.1 The OR operator: |
		B.4.2 Groups: ()
		B.4.3 Character classes: []
	B.5 Style
	B.6 Mastery
appendix C Vectors and linear algebra
	C.1 Vectors
		C.1.1 Vector length and normalization
		C.1.2 Comparing vectors
		C.1.3 Dot product
	C.2 Matrices
		C.2.1 Multiplying matrices
		C.2.2 Matrices as transformations
	C.3 Tensors
	C.4 Diving deeper into linear algebra
appendix D Machine learning tools and techniques
	D.1 Change of paradigm
		D.1.1 Machine learning workflow
		D.1.2 Problem definition
		D.1.3 Data selection and avoiding bias
	D.2 How fit is “fit”?
		D.2.1 Knowing is half the battle
		D.2.2 Cross-validation
		D.2.3 Imbalanced training sets
	D.3 Model optimization
	D.4 Model evaluation and performance metrics
		D.4.1 Measuring classifier performance
		D.4.2 Measuring regressor performance
	D.5 Pro tips
appendix E Deploying NLU containerized microservices
	E.1 A multilabel intent classifier
	E.2 Problem statement and training data
	E.3 Microservices architecture
		E.3.1 Containerizing a microservice
		E.3.2 Building a container from a repository
		E.3.3 Scaling an NLU service
	E.4 Running and testing the prediction microservice
		E.4.1 Setting up DigitalOcean Spaces and training the model
		E.4.2 Downloading the nlu-fastapi container image
		E.4.3 Running the container
		E.4.4 Interacting with the container API
appendix F Glossary
	F.1 Acronyms
	F.2 Terms
notes
	Chapter 1
	Chapter 2
	Chapter 3
	Chapter 4
	Chapter 5
	Chapter 6
	Chapter 7
	Chapter 8
	Chapter 9
	Chapter 10
	Chapter 11
	Chapter 12
	Appendix A
	Appendix B
	Appendix C
	Appendix D
	Appendix E
	Appendix F
index
	Symbols
	Numerics
	A
	B
	C
	D
	E
	F
	G
	H
	I
	J
	K
	L
	M
	N
	O
	P
	Q
	R
	S
	T
	U
	V
	W
	Z