Getting Started with Google BERT: Build and train state-of-the-art natural language processing models using BERT

Cover
Title Page
Copyright and Credits
Dedication
About Packt
Contributors
Table of Contents
Preface
Section 1 - Starting Off with BERT
Chapter 1: A Primer on Transformers
	Introduction to the transformer 
	Understanding the encoder of the transformer 
		Self-attention mechanism 
			Understanding the self-attention mechanism 
				Step 1
				Step 2
				Step 3
				Step 4
		Multi-head attention mechanism 
		Learning position with positional encoding 
		Feedforward network
		Add and norm component 
		Putting all the encoder components together 
	Understanding the decoder of a transformer
		Masked multi-head attention 
		Multi-head attention 
		Feedforward network
		Add and norm component 
		Linear and softmax layers
		Putting all the decoder components together 
	Putting the encoder and decoder together 
	Training the transformer
	Summary
	Questions
	Further reading
Chapter 2: Understanding the BERT Model
	Basic idea of BERT 
	Working of BERT 
	Configurations of BERT 
		BERT-base
		BERT-large
		Other configurations of BERT
	Pre-training the BERT model
		Input data representation
			Token embedding 
			Segment embedding
			Position embedding 
			Final representation 
			WordPiece tokenizer 
		Pre-training strategies 
			Language modeling
				Auto-regressive language modeling 
				Auto-encoding language modeling
			Masked language modeling
				Whole word masking 
			Next sentence prediction 
		Pre-training procedure 
	Subword tokenization algorithms 
		Byte pair encoding 
			Tokenizing with BPE 
		Byte-level byte pair encoding 
		WordPiece
	Summary
	Questions
	Further reading
Chapter 3: Getting Hands-On with BERT
	Exploring the pre-trained BERT model
	Extracting embeddings from pre-trained BERT 
		Hugging Face transformers 
		Generating BERT embeddings
			Preprocessing the input 
			Getting the embedding 
	Extracting embeddings from all encoder layers of BERT
		Extracting the embeddings 
			Preprocessing the input
			Getting the embeddings 
	Fine-tuning BERT for downstream tasks
		Text classification 
			Fine-tuning BERT for sentiment analysis 
				Importing the dependencies 
				Loading the model and dataset
				Preprocessing the dataset
				Training the model 
		Natural language inference 
		Question-answering
			Performing question-answering with fine-tuned BERT 
				Preprocessing the input
				Getting the answer
		Named entity recognition 
	Summary 
	Questions
	Further reading 
Section 2 - Exploring BERT Variants
Chapter 4: BERT Variants I - ALBERT, RoBERTa, ELECTRA, and SpanBERT
	A Lite version of BERT 
		Cross-layer parameter sharing 
		Factorized embedding parameterization
		Training the ALBERT model
			Sentence order prediction
		Comparing ALBERT with BERT 
	Extracting embeddings with ALBERT
	Robustly Optimized BERT pre-training Approach
		Using dynamic masking instead of static masking 
		Removing the NSP task
		Training with more data points
		Training with a large batch size 
		Using BBPE as a tokenizer 
			Exploring the RoBERTa tokenizer 
	Understanding ELECTRA 
		Understanding the replaced token detection task 
		Exploring the generator and discriminator of ELECTRA 
		Training the ELECTRA model
		Exploring efficient training methods
	Predicting span with SpanBERT
		Understanding the architecture of SpanBERT
		Exploring SpanBERT 
		Performing Q&As with pre-trained SpanBERT 
	Summary
	Questions
	Further reading 
Chapter 5: BERT Variants II - Based on Knowledge Distillation
	Introducing knowledge distillation 
		Training the student network 
	DistilBERT – the distilled version of BERT 
		Teacher-student architecture 
			The teacher BERT
			The student BERT
		Training the student BERT (DistilBERT) 
	Introducing TinyBERT 
		Teacher-student architecture  
			Understanding the teacher BERT  
			Understanding the student BERT 
		Distillation in TinyBERT 
			Transformer layer distillation 
				Attention-based distillation
				Hidden state-based distillation 
			Embedding layer distillation 
			Prediction layer distillation
		The final loss function 
		Training the student BERT (TinyBERT)
			General distillation 
			Task-specific distillation 
				The data augmentation method 
	Transferring knowledge from BERT to neural networks
		Teacher-student architecture 
			The teacher BERT 
			The student network 
		Training the student network  
		The data augmentation method
			Understanding the masking method
			Understanding the POS-guided word replacement method 
			Understanding the n-gram sampling method
			The data augmentation procedure
	Summary
	Questions
	Further reading 
Section 3 - Applications of BERT
Chapter 6: Exploring BERTSUM for Text Summarization
	Text summarization 
		Extractive summarization
		Abstractive summarization 
	Fine-tuning BERT for text summarization 
		Extractive summarization using BERT 
			BERTSUM with a classifier 
			BERTSUM with a transformer and LSTM 
				BERTSUM with an inter-sentence transformer 
				BERTSUM with LSTM 
		Abstractive summarization using BERT 
	Understanding ROUGE evaluation metrics
		Understanding the ROUGE-N metric 
			ROUGE-1 
			ROUGE-2 
		Understanding ROUGE-L  
	The performance of the BERTSUM model 
	Training the BERTSUM model 
	Summary 
	Questions
	Further reading
Chapter 7: Applying BERT to Other Languages
	Understanding multilingual BERT 
		Evaluating M-BERT on the NLI task 
			Zero-shot 
			TRANSLATE-TEST 
			TRANSLATE-TRAIN
			TRANSLATE-TRAIN-ALL
	How multilingual is multilingual BERT? 
		Effect of vocabulary overlap
		Generalization across scripts 
		Generalization across typological features 
		Effect of language similarity
		Effect of code switching and transliteration
			Code switching 
			Transliteration 
			M-BERT on code switching and transliteration 
	The cross-lingual language model
		Pre-training strategies 
			Causal language modeling 
			Masked language modeling 
			Translation language modeling 
		Pre-training the XLM model
		Evaluation of XLM
	Understanding XLM-R
	Language-specific BERT 
		FlauBERT for French 
			Getting a representation of a French sentence with FlauBERT 
			French Language Understanding Evaluation
		BETO for Spanish 
			Predicting masked words using BETO 
		BERTje for Dutch
			Next sentence prediction with BERTje
		German BERT 
		Chinese BERT 
		Japanese BERT 
		FinBERT for Finnish
		UmBERTo for Italian 
		BERTimbau for Portuguese 
		RuBERT for Russian 
	Summary
	Questions
	Further reading
Chapter 8: Exploring Sentence and Domain-Specific BERT
	Learning about sentence representation with Sentence-BERT  
		Computing sentence representation 
		Understanding Sentence-BERT 
			Sentence-BERT with a Siamese network 
				Sentence-BERT for a sentence pair classification task
				Sentence-BERT for a sentence pair regression task
			Sentence-BERT with a triplet network
	Exploring the sentence-transformers library 
		Computing sentence representation using Sentence-BERT 
		Computing sentence similarity 
		Loading custom models
		Finding a similar sentence with Sentence-BERT 
	Learning multilingual embeddings through knowledge distillation 
		Teacher-student architecture
		Using the multilingual model 
	Domain-specific BERT 
		ClinicalBERT 
			Pre-training ClinicalBERT 
			Fine-tuning ClinicalBERT 
			Extracting clinical word similarity 
		BioBERT 
			Pre-training the BioBERT model
			Fine-tuning the BioBERT model 
				BioBERT for NER tasks 
				BioBERT for question answering 
	Summary 
	Questions
	Further reading
Chapter 9: Working with VideoBERT, BART, and More
	Learning language and video representations with VideoBERT 
		Pre-training a VideoBERT model  
			Cloze task 
			Linguistic-visual alignment  
			The final pre-training objective 
		Data source and preprocessing 
		Applications of VideoBERT 
			Predicting the next visual tokens
			Text-to-video generation 
			Video captioning 
	Understanding BART 
		Architecture of BART 
			Noising techniques 
				Token masking
				Token deletion
				Token infilling 
				Sentence shuffling 
				Document rotation
		Comparing different pre-training objectives 
		Performing text summarization with BART 
	Exploring BERT libraries 
		Understanding ktrain
			Sentiment analysis using ktrain
			Building a document answering model 
			Document summarization 
		bert-as-service 
			Installing the library 
			Computing sentence representation
			Computing contextual word representation 
	Summary 
	Questions 
	Further reading
Assessments
	Chapter 1, A Primer on Transformers
	Chapter 2, Understanding the BERT Model
	Chapter 3, Getting Hands-On with BERT
	Chapter 4, BERT Variants I – ALBERT, RoBERTa, ELECTRA, SpanBERT
	Chapter 5, BERT Variants II – Based on Knowledge Distillation
	Chapter 6, Exploring BERTSUM for Text Summarization
	Chapter 7, Applying BERT to Other Languages
	Chapter 8, Exploring Sentence- and Domain-Specific BERT
	Chapter 9, Working with VideoBERT, BART, and More
Other Books You May Enjoy
Index