Transfer Learning for Natural Language Processing

Transfer Learning for Natural Language Processing
contents
preface
acknowledgments
about this book
	Who should read this book?
	Road map
	Software requirements
	About the code
	liveBook discussion forum
about the author
about the cover illustration
Part 1—Introduction and overview
	1 What is transfer learning?
		1.1 Overview of representative NLP tasks
		1.2 Understanding NLP in the context of AI
			1.2.1 Artificial intelligence (AI)
			1.2.2 Machine learning
			1.2.3 Natural language processing (NLP)
		1.3 A brief history of NLP advances
			1.3.1 General overview
			1.3.2 Recent transfer learning advances
		1.4 Transfer learning in computer vision
			1.4.1 General overview
			1.4.2 Pretrained ImageNet models
			1.4.3 Fine-tuning pretrained ImageNet models
		1.5 Why is NLP transfer learning an exciting topic to study now?
		Summary
	2 Getting started with baselines: Data preprocessing
		2.1 Preprocessing email spam classification example data
			2.1.1 Loading and visualizing the Enron corpus
			2.1.2 Loading and visualizing the fraudulent email corpus
			2.1.3 Converting the email text into numbers
		2.2 Preprocessing movie sentiment classification example data
		2.3 Generalized linear models
			2.3.1 Logistic regression
			2.3.2 Support vector machines (SVMs)
		Summary
	3 Getting started with baselines: Benchmarking and optimization
		3.1 Decision-tree-based models
			3.1.1 Random forests (RFs)
			3.1.2 Gradient-boosting machines (GBMs)
		3.2 Neural network models
			3.2.1 Embeddings from Language Models (ELMo)
			3.2.2 Bidirectional Encoder Representations from Transformers (BERT)
		3.3 Optimizing performance
			3.3.1 Manual hyperparameter tuning
			3.3.2 Systematic hyperparameter tuning
		Summary
Part 2—Shallow transfer learning and deep transfer learning with recurrent neural networks (RNNs)
	4 Shallow transfer learning for NLP
		4.1 Semisupervised learning with pretrained word embeddings
		4.2 Semisupervised learning with higher-level representations
		4.3 Multitask learning
			4.3.1 Problem setup and a shallow neural single-task baseline
			4.3.2 Dual-task experiment
		4.4 Domain adaptation
		Summary
	5 Preprocessing data for recurrent neural network deep transfer learning experiments
		5.1 Preprocessing tabular column-type classification data
			5.1.1 Obtaining and visualizing tabular data
			5.1.2 Preprocessing tabular data
			5.1.3 Encoding preprocessed data as numbers
		5.2 Preprocessing fact-checking example data
			5.2.1 Special problem considerations
			5.2.2 Loading and visualizing fact-checking data
		Summary
	6 Deep transfer learning for NLP with recurrent neural networks
		6.1 Semantic Inference for the Modeling of Ontologies (SIMOn)
			6.1.1 General neural architecture overview
			6.1.2 Modeling tabular data
			6.1.3 Application of SIMOn to tabular column-type classification data
		6.2 Embeddings from Language Models (ELMo)
			6.2.1 ELMo bidirectional language modeling
			6.2.2 Application to fake news detection
		6.3 Universal Language Model Fine-Tuning (ULMFiT)
			6.3.1 Target task language model fine-tuning
			6.3.2 Target task classifier fine-tuning
		Summary
Part 3—Deep transfer learning with transformers and adaptation strategies
	7 Deep transfer learning for NLP with the transformer and GPT
		7.1 The transformer
			7.1.1 An introduction to the transformers library and attention visualization
			7.1.2 Self-attention
			7.1.3 Residual connections, encoder-decoder attention, and positional encoding
			7.1.4 Application of pretrained encoder-decoder to translation
		7.2 The Generative Pretrained Transformer
			7.2.1 Architecture overview
			7.2.2 Transformers pipelines introduction and application to text generation
			7.2.3 Application to chatbots
		Summary
	8 Deep transfer learning for NLP with BERT and multilingual BERT
		8.1 Bidirectional Encoder Representations from Transformers (BERT)
			8.1.1 Model architecture
			8.1.2 Application to question answering
			8.1.3 Application to fill in the blanks and next-sentence prediction tasks
		8.2 Cross-lingual learning with multilingual BERT (mBERT)
			8.2.1 Brief JW300 dataset overview
			8.2.2 Transfer mBERT to monolingual Twi data with the pretrained tokenizer
			8.2.3 mBERT and tokenizer trained from scratch on monolingual Twi data
		Summary
	9 ULMFiT and knowledge distillation adaptation strategies
		9.1 Gradual unfreezing and discriminative fine-tuning
			9.1.1 Pretrained language model fine-tuning
			9.1.2 Target task classifier fine-tuning
		9.2 Knowledge distillation
			9.2.1 Transfer DistilmBERT to monolingual Twi data with pretrained tokenizer
		Summary
	10 ALBERT, adapters, and multitask adaptation strategies
		10.1 Embedding factorization and cross-layer parameter sharing
			10.1.1 Fine-tuning pretrained ALBERT on MDSD book reviews
		10.2 Multitask fine-tuning
			10.2.1 General Language Understanding Dataset (GLUE)
			10.2.2 Fine-tuning on a single GLUE task
			10.2.3 Sequential adaptation
		10.3 Adapters
		Summary
	11 Conclusions
		11.1 Overview of key concepts
		11.2 Other emerging research trends
			11.2.1 RoBERTa
			11.2.2 GPT-3
			11.2.3 XLNet
			11.2.4 BigBird
			11.2.5 Longformer
			11.2.6 Reformer
			11.2.7 T5
			11.2.8 BART
			11.2.9 XLM
			11.2.10 TAPAS
		11.3 Future of transfer learning in NLP
		11.4 Ethical and environmental considerations
		11.5 Staying up-to-date
			11.5.1 Kaggle and Zindi competitions
			11.5.2 arXiv
			11.5.3 News and social media (Twitter)
		11.6 Final words
		Summary
Appendix A—Kaggle primer
	A.1 Free GPUs with Kaggle kernels
	A.2 Competitions, discussion, and blog
Appendix B—Introduction to fundamental deep learning tools
	B.1 Stochastic gradient descent
	B.2 TensorFlow
	B.3 PyTorch
	B.4 Keras, fast.ai, and Transformers by Hugging Face
index
	A
	B
	C
	D
	E
	F
	G
	H
	I
	J
	K
	L
	M
	N
	O
	P
	Q
	R
	S
	T
	U
	V
	W
	X
	Z