Mastering Transformers: Build state-of-the-art models from scratch with advanced natural language processing techniques

Cover
Title page
Copyright and Credits
Contributors
Table of Contents
Preface
Section 1: Introduction – Recent Developments in the Field, Installations, and Hello World Applications
Chapter 1: From Bag-of-Words to the Transformer
	Technical requirements
	Evolution of NLP toward Transformers
	Understanding distributional semantics
		BoW implementation
		Overcoming the dimensionality problem
		Language modeling and generation
	Leveraging DL
		Learning word embeddings
		A brief overview of RNNs
		LSTMs and gated recurrent units
		A brief overview of CNNs
	Overview of the Transformer architecture
		Attention mechanism
		Multi-head attention mechanisms
	Using TL with Transformers
	Summary
	References
Chapter 2: A Hands-On Introduction to the Subject
	Technical requirements
	Installing Transformer with Anaconda
		Installation on Linux
		Installation on Windows
		Installation on macOS
		Installing TensorFlow, PyTorch, and Transformer
		Installing using Google Colab
	Working with language models and tokenizers
	Working with community-provided models
	Working with benchmarks and datasets
		Important benchmarks
		Accessing the datasets with an Application Programming Interface
	Benchmarking for speed and memory
	Summary
Section 2: Transformer Models – From Autoencoding to Autoregressive Models
Chapter 3: Autoencoding Language Models
	Technical requirements
	BERT – one of the autoencoding language models
		BERT language model pretraining tasks
		A deeper look into the BERT language model
	Autoencoding language model training for any language
	Sharing models with the community
	Understanding other autoencoding models
		Introducing ALBERT
		RoBERTa
		ELECTRA
	Working with tokenization algorithms
		Byte pair encoding
		WordPiece tokenization
		Sentence piece tokenization
		The tokenizers library
	Summary
Chapter 4: Autoregressive and Other Language Models
	Technical requirements
	Working with AR language models
		Introduction and training models with GPT
		Transformer-XL
		XLNet
	Working with Seq2Seq models
		T5
		Introducing BART
	AR language model training
	NLG using AR models
	Summarization and MT fine-tuning using simpletransformers
	Summary
	References
Chapter 5: Fine-Tuning Language Models for Text Classification
	Technical requirements
	Introduction to text classification
	Fine-tuning a BERT model for single-sentence binary classification
	Training a classification model with native PyTorch
	Fine-tuning BERT for multi-class classification with custom datasets
	Fine-tuning the BERT model for sentence-pair regression
	Utilizing run_glue.py to fine-tune the models
	Summary
Chapter 6: Fine-Tuning Language Models for Token Classification
	Technical requirements
	Introduction to token classification
		Understanding NER
		Understanding POS tagging
		Understanding QA
	Fine-tuning language models for NER
	Question answering using token classification
	Summary
Chapter 7: Text Representation
	Technical requirements
	Introduction to sentence embeddings
		Cross-encoder versus bi-encoder
		Benchmarking sentence similarity models
		Using BART for zero-shot learning
	Semantic similarity experiment with FLAIR
		Average word embeddings
		RNN-based document embeddings
		Transformer-based BERT embeddings
		Sentence-BERT embeddings
	Text clustering with Sentence-BERT
		Topic modeling with BERTopic
	Semantic search with Sentence-BERT
	Summary
	Further reading
Section 3: Advanced Topics
Chapter 8: Working with Efficient Transformers
	Technical requirements
	Introduction to efficient, light, and fast transformers
	Implementation for model size reduction
		Working with DistilBERT for knowledge distillation
		Pruning transformers
		Quantization
	Working with efficient self-attention
		Sparse attention with fixed patterns
		Learnable patterns
		Low-rank factorization, kernel methods, and other approaches
	Summary
	References
Chapter 9: Cross-Lingual and Multilingual Language Modeling
	Technical requirements
	Translation language modeling and cross-lingual knowledge sharing
	XLM and mBERT
		mBERT
		XLM
	Cross-lingual similarity tasks
		Cross-lingual text similarity
		Visualizing cross-lingual textual similarity
	Cross-lingual classification
	Cross-lingual zero-shot learning
	Fundamental limitations of multilingual models
		Fine-tuning the performance of multilingual models
	Summary
	References
Chapter 10: Serving Transformer Models
	Technical requirements
	fastAPI Transformer model serving
	Dockerizing APIs
	Faster Transformer model serving using TFX
	Load testing using Locust
	Summary
	References
Chapter 11: Attention Visualization and Experiment Tracking
	Technical requirements
	Interpreting attention heads
		Visualizing attention heads with exBERT
		Multiscale visualization of attention heads with BertViz
		Understanding the inner parts of BERT with probing classifiers
	Tracking model metrics
		Tracking model training with TensorBoard
		Tracking model training live with W&B
	Summary
	References
	Why subscribe?
About Packt
Other Books You May Enjoy
Index