Transformers for Natural Language Processing: Build innovative deep neural network architectures for NLP with Python, PyTorch, TensorFlow, BERT, RoBERTa, and more

Cover
Copyright
Packt Page
Contributors
Table of Contents
Preface
Chapter 1: Getting Started with the Model Architecture of the Transformer
	The background of the Transformer
	The rise of the Transformer: Attention Is All You Need
		The encoder stack
			Input embedding
			Positional encoding
			Sub-layer 1: Multi-head attention
			Sub-layer 2: Feedforward network
		The decoder stack
			Output embedding and position encoding
			The attention layers
			The FFN sub-layer, the Post-LN, and the linear layer
	Training and performance
		Before we end the chapter
	Summary
	Questions
	References
Chapter 2: Fine-Tuning BERT Models
	The architecture of BERT
		The encoder stack
			Preparing the pretraining input environment
		Pretraining and fine-tuning a BERT model
	Fine-tuning BERT
		Activating the GPU
		Installing the Hugging Face PyTorch interface for BERT
		Importing the modules
		Specifying CUDA as the device for torch
		Loading the dataset
		Creating sentences, label lists, and adding BERT tokens
		Activating the BERT tokenizer
		Processing the data
		Creating attention masks
		Splitting data into training and validation sets
		Converting all the data into torch tensors
		Selecting a batch size and creating an iterator
		BERT model configuration
		Loading the Hugging Face BERT uncased base model
		Optimizer grouped parameters
		The hyperparameters for the training loop
		The training loop
		Training evaluation
		Predicting and evaluating using the holdout dataset
		Evaluating using Matthews Correlation Coefficient
		The score of individual batches
		Matthews evaluation for the whole dataset
	Summary
	Questions
	References
Chapter 3: Pretraining a RoBERTa Model from Scratch
	Training a tokenizer and pretraining a transformer
	Building KantaiBERT from scratch
		Step 1: Loading the dataset
		Step 2: Installing Hugging Face transformers
		Step 3: Training a tokenizer
		Step 4: Saving the files to disk
		Step 5: Loading the trained tokenizer files
		Step 6: Checking resource constraints: GPU and CUDA
		Step 7: Defining the configuration of the model
		Step 8: Reloading the tokenizer in transformers
		Step 9: Initializing a model from scratch
			Exploring the parameters
		Step 10: Building the dataset
		Step 11: Defining a data collator
		Step 12: Initializing the trainer
		Step 13: Pretraining the model
		Step 14: Saving the final model (+tokenizer + config) to disk
		Step 15: Language modeling with FillMaskPipeline
	Next steps
	Summary
	Questions
	References
Chapter 4: Downstream NLP Tasks with Transformers
	Transduction and the inductive inheritance of transformers
		The human intelligence stack
		The machine intelligence stack
	Transformer performances versus Human Baselines
		Evaluating models with metrics
			Accuracy score
			F1-score
			Matthews Correlation Coefficient (MCC)
		Benchmark tasks and datasets
			From GLUE to SuperGLUE
			Introducing higher Human Baseline standards
			The SuperGLUE evaluation process
		Defining the SuperGLUE benchmark tasks
			BoolQ
			Commitment Bank (CB)
			Multi-Sentence Reading Comprehension (MultiRC)
			Reading Comprehension with Commonsense Reasoning Dataset (ReCoRD)
			Recognizing Textual Entailment (RTE)
			Words in Context (WiC)
			The Winograd Schema Challenge (WSC)
	Running downstream tasks
		The Corpus of Linguistic Acceptability (CoLA)
		Stanford Sentiment TreeBank (SST-2)
		Microsoft Research Paraphrase Corpus (MRPC)
		Winograd schemas
	Summary
	Questions
	References
Chapter 5: Machine Translation with the Transformer
	Defining machine translation
		Human transductions and translations
		Machine transductions and translations
	Preprocessing a WMT dataset
		Preprocessing the raw data
		Finalizing the preprocessing of the datasets
	Evaluating machine translation with BLEU
		Geometric evaluations
		Applying a smoothing technique
			Chencherry smoothing
	Translations with Trax
		Installing Trax
		Creating a Transformer model
		Initializing the model using pretrained weights
		Tokenizing a sentence
		Decoding from the Transformer
		De-tokenizing and displaying the translation
	Summary
	Questions
	References
Chapter 6: Text Generation with OpenAI GPT-2 and GPT-3 Models
	The rise of billion-parameter transformer models
		The increasing size of transformer models
			Context size and maximum path length
	Transformers, reformers, PET, or GPT?
		The limits of the original Transformer architecture
			Running BertViz
		The Reformer
		Pattern-Exploiting Training (PET)
			The philosophy of Pattern-Exploiting Training (PET)
	It\'s time to make a decision
	The architecture of OpenAI GPT models
		From fine-tuning to zero-shot models
		Stacking decoder layers
	Text completion with GPT-2
		Step 1: Activating the GPU
		Step 2: Cloning the OpenAI GPT-2 repository
		Step 3: Installing the requirements
		Step 4: Checking the version of TensorFlow
		Step 5: Downloading the 345M parameter GPT-2 model
		Steps 6-7: Intermediate instructions
		Steps 7b-8: Importing and defining the model
		Step 9: Interacting with GPT-2
	Training a GPT-2 language model
		Step 1: Prerequisites
		Steps 2 to 6: Initial steps of the training process
		Step 7: The N Shepperd training files
		Step 8: Encoding the dataset
		Step 9: Training the model
		Step 10: Creating a training model directory
	Context and completion examples
	Generating music with transformers
	Summary
	Questions
	References
Chapter 7: Applying Transformers to Legal and Financial Documents for AI Text Summarization
	Designing a universal text-to-text model
		The rise of text-to-text transformer models
		A prefix instead of task-specific formats
		The T5 model
	Text summarization with T5
		Hugging Face
			Hugging Face transformer resources
		Initializing the T5-large transformer model
			Getting started with T5
			Exploring the architecture of the T5 model
		Summarizing documents with T5-large
			Creating a summarization function
			A general topic sample
			The Bill of Rights sample
			A corporate law sample
	Summary
	Questions
	References
Chapter 8: Matching Tokenizers and Datasets
	Matching datasets and tokenizers
		Best practices
			Step 1: Preprocessing
			Step 2: Post-processing
			Continuous human quality control
		Word2Vec tokenization
			Case 0: Words in the dataset and the dictionary
			Case 1: Words not in the dataset or the dictionary
			Case 2: Noisy relationships
			Case 3: Rare words
			Case 4: Replacing rare words
			Case 5: Entailment
	Standard NLP tasks with specific vocabulary
		Generating unconditional samples with GPT-2
			Controlling tokenized data
		Generating trained conditional samples
	T5 Bill of Rights Sample
		Summarizing the Bill of Rights, version 1
		Summarizing the Bill of Rights, version 2
	Summary
	Questions
	References
Chapter 9: Semantic Role Labeling with BERT-Based Transformers
	Getting started with SRL
		Defining Semantic Role Labeling
			Visualizing SRL
		Running a pretrained BERT-based model
			The architecture of the BERT-based model
			Setting up the BERT SRL environment
	SRL experiments with the BERT-based model
	Basic samples
		Sample 1
		Sample 2
		Sample 3
	Difficult samples
		Sample 4
		Sample 5
		Sample 6
	Summary
	Questions
	References
Chapter 10: Let Your Data Do the Talking: Story, Questions, and Answers
	Methodology
		Transformers and methods
	Method 0: Trial and error
	Method 1: NER first
		Using NER to find questions
			Location entity questions
			Person entity questions
	Method 2: SRL first
		Question-answering with ELECTRA
		Project management constraints
		Using SRL to find questions
	Next steps
		Exploring Haystack with a RoBERTa model
	Summary
	Questions
	References
Chapter 11: Detecting Customer Emotions to Make Predictions
	Getting started: Sentiment analysis transformers
	The Stanford Sentiment Treebank (SST)
		Sentiment analysis with RoBERTa-large
	Predicting customer behavior with sentiment analysis
		Sentiment analysis with DistilBERT
		Sentiment analysis with Hugging Face\'s models list
			DistilBERT for SST
			MiniLM-L12-H384-uncased
			RoBERTa-large-mnli
			BERT-base multilingual model
	Summary
	Questions
	References
Chapter 12: Analyzing Fake News with Transformers
	Emotional reactions to fake news
		Cognitive dissonance triggers emotional reactions
			Analyzing a conflictual Tweet
			Behavioral representation of fake news
	A rational approach to fake news
		Defining a fake news resolution roadmap
		Gun control
			Sentiment analysis
			Named entity recognition (NER)
			Semantic role labeling (SRL)
			Reference sites
		COVID-19 and former President Trump\'s Tweets
			Semantic Role Labeling (SRL)
	Before we go
		Looking for the silver bullet
		Looking for reliable training methods
	Summary
	Questions
	References
Appendix: Answers to the Questions
	Chapter 1, Getting Started with the Model Architecture of the Transformer
	Chapter 2, Fine-Tuning BERT Models
	Chapter 3, Pretraining a RoBERTa Model from Scratch
	Chapter 4, Downstream NLP Tasks with Transformers
	Chapter 5, Machine Translation with the Transformer
	Chapter 6, Text Generation with OpenAI GPT-2 and GPT-3 Models
	Chapter 7, Applying Transformers to Legal and Financial Documents for AI Text Summarization
	Chapter 8, Matching Tokenizers and Datasets
	Chapter 9, Semantic Role Labeling with BERT-Based Transformers
	Chapter 10, Let Your Data Do the Talking: Story, Questions, and Answers
	Chapter 11, Detecting Customer Emotions to Make Predictions
	Chapter 12, Analyzing Fake News with Transformers
Other Books You May Enjoy
Index