Foundation Models for Natural Language Processing: Pre-trained Language Models Integrating Media

Foreword
Preface
	Acknowledgments
Contents
About the Authors
1 Introduction
	1.1 Scope of the Book
	1.2 Preprocessing of Text
	1.3 Vector Space Models and Document Classification
	1.4 Nonlinear Classifiers
	1.5 Generating Static Word Embeddings
	1.6 Recurrent Neural Networks
	1.7 Convolutional Neural Networks
	1.8 Summary
	References
2 Pre-trained Language Models
	2.1 BERT: Self-Attention and Contextual Embeddings
		2.1.1 BERT Input Embeddings and Self-Attention
			Self-Attention to Generate Contextual Embeddings
		2.1.2 Training BERT by Predicting Masked Tokens
		2.1.3 Fine-Tuning BERT to Downstream Tasks
		2.1.4 Visualizing Attentions and Embeddings
		2.1.5 Natural Language Understanding by BERT
			BERT's Performance on Other Fine-Tuning Tasks
		2.1.6 Computational Complexity
		2.1.7 Summary
	2.2 GPT: Autoregressive Language Models
		2.2.1 The Task of Autoregressive Language Models
		2.2.2 Training GPT by Predicting the Next Token
			Visualizing GPT Embeddings
		2.2.3 Generating a Sequence of Words
		2.2.4 The Advanced Language Model GPT-2
		2.2.5 Fine-Tuning GPT
		2.2.6 Summary
	2.3 Transformer: Sequence-to-Sequence Translation
		2.3.1 The Transformer Architecture
			Cross-Attention
		2.3.2 Decoding a Translation to Generate the Words
		2.3.3 Evaluation of a Translation
		2.3.4 Pre-trained Language Models and Foundation Models
			Available Implementations
		2.3.5 Summary
	2.4 Training and Assessment of Pre-trained Language Models
		2.4.1 Optimization of PLMs
			Basics of PLM Optimization
			Variants of Stochastic Gradient Descent
			Parallel Training for Large Models
		2.4.2 Regularization of Pre-trained Language Models
		2.4.3 Neural Architecture Search
		2.4.4 The Uncertainty of Model Predictions
			Bayesian Neural Networks
			Estimating Uncertainty by a Single Deterministic Model
			Representing the Predictive Distribution by Ensembles
		2.4.5 Explaining Model Predictions
			Linear Local Approximations
			Nonlinear Local Approximations
			Explanation by Retrieval
			Explanation by Generating a Chain of Thought
		2.4.6 Summary
	References
3 Improving Pre-trained Language Models
	3.1 Modifying Pre-training Objectives
		3.1.1 Autoencoders Similar to BERT
		3.1.2 Autoregressive Language Models Similar to GPT
		3.1.3 Transformer Encoder-Decoders
		3.1.4 Systematic Comparison of Transformer Variants
		3.1.5 Summary
	3.2 Capturing Longer Dependencies
		3.2.1 Sparse Attention Matrices
		3.2.2 Hashing and Low-Rank Approximations
		3.2.3 Comparisons of Transformers with Long Input Sequences
		3.2.4 Summary
	3.3 Multilingual Pre-trained Language Models
		3.3.1 Autoencoder Models
		3.3.2 Seq2seq Transformer Models
		3.3.3 Autoregressive Language Models
		3.3.4 Summary
	3.4 Additional Knowledge for Pre-trained Language Models
		3.4.1 Exploiting Knowledge Base Embeddings
		3.4.2 Pre-trained Language Models for Graph Learning
		3.4.3 Textual Encoding of Tables
		3.4.4 Textual Encoding of Knowledge Base Relations
		3.4.5 Enhancing Pre-trained Language Models by Retrieved Texts
		3.4.6 Summary
	3.5 Changing Model Size
		3.5.1 Larger Models Usually Have a better Performance
		3.5.2 Mixture-of-Experts Models
		3.5.3 Parameter Compression and Reduction
		3.5.4 Low-Rank Factorization
		3.5.5 Knowledge Distillation
		3.5.6 Summary
	3.6 Fine-Tuning for Specific Applications
		3.6.1 Properties of Fine-Tuning
			Catastrophic Forgetting
			Fine-Tuning and Overfitting
		3.6.2 Fine-Tuning Variants
			Fine-Tuning in Two Stages
			Fine-Tuning for Multiple Tasks
			Meta-Learning to Accelerate Fine-Tuning
			Fine-Tuning a Frozen Model by Adapters
			Fine-Tuning GPT-3
		3.6.3 Creating Few-Shot Prompts
		3.6.4 Thought Chains for Few-Shot Learning of Reasoning
		3.6.5 Fine-Tuning Models to Execute Instructions
			InstructGPT Results
			Instruction Tuning with FLAN
		3.6.6 Generating Labeled Data by Foundation Models
		3.6.7 Summary
	References
4 Knowledge Acquired by Foundation Models
	4.1 Benchmark Collections
		4.1.1 The GLUE Benchmark Collection
		4.1.2 SuperGLUE: An Advanced Version of GLUE
		4.1.3 Text Completion Benchmarks
		4.1.4 Large Benchmark Collections
		4.1.5 Summary
	4.2 Evaluating Knowledge by Probing Classifiers
		4.2.1 BERT's Syntactic Knowledge
		4.2.2 Common Sense Knowledge
		4.2.3 Logical Consistency
			Improving Logical Consistency
		4.2.4 Summary
	4.3 Transferability and Reproducibility of Benchmarks
		4.3.1 Transferability of Benchmark Results
			Benchmarks May Not Test All Aspects
			Logical Reasoning by Correlation
		4.3.2 Reproducibility of Published Results in Natural Language Processing
			Available Implementations
		4.3.3 Summary
	References
5 Foundation Models for Information Extraction
	5.1 Text Classification
		5.1.1 Multiclass Classification with Exclusive Classes
		5.1.2 Multilabel Classification
		5.1.3 Few- and Zero-Shot Classification
			Available Implementations
		5.1.4 Summary
	5.2 Word Sense Disambiguation
		5.2.1 Sense Inventories
		5.2.2 Models
			Available Implementations
		5.2.3 Summary
	5.3 Named Entity Recognition
		5.3.1 Flat Named Entity Recognition
		5.3.2 Nested Named Entity Recognition
			Available Implementations
		5.3.3 Entity Linking
			Available Implementations
		5.3.4 Summary
	5.4 Relation Extraction
		5.4.1 Coreference Resolution
			Available Implementations
		5.4.2 Sentence-Level Relation Extraction
		5.4.3 Document-Level Relation Extraction
		5.4.4 Joint Entity and Relation Extraction
			Aspect-Based Sentiment Analysis
			Semantic Role Labeling
			Extracting Knowledge Graphs from Pre-trained PLMs
		5.4.5 Distant Supervision
		5.4.6 Relation Extraction Using Layout Information
			Available Implementations
		5.4.7 Summary
	References
6 Foundation Models for Text Generation
	6.1 Document Retrieval
		6.1.1 Dense Retrieval
		6.1.2 Measuring Text Retrieval Performance
		6.1.3 Cross-Encoders with BERT
		6.1.4 Using Token Embeddings for Retrieval
		6.1.5 Dense Passage Embeddings and Nearest Neighbor Search
			Available Implementations
		6.1.6 Summary
	6.2 Question Answering
		6.2.1 Question Answering Based on Training Data Knowledge
			Fine-Tuned Question Answering Models
			Question Answering with Few-Shot Language Models
		6.2.2 Question Answering Based on Retrieval
		6.2.3 Long-Form Question Answering Using Retrieval
			A Language Model with Integrated Retrieval
			Controlling a Search Engine by a Pre-trained Language Model
			Available Implementations
		6.2.4 Summary
	6.3 Neural Machine Translation
		6.3.1 Translation for a Single Language Pair
		6.3.2 Multilingual Translation
		6.3.3 Multilingual Question Answering
			Available Implementations
		6.3.4 Summary
	6.4 Text Summarization
		6.4.1 Shorter Documents
		6.4.2 Longer Documents
		6.4.3 Multi-Document Summarization
			Available Implementations
		6.4.4 Summary
	6.5 Text Generation
		6.5.1 Generating Text by Language Models
		6.5.2 Generating Text with a Given Style
			Style-Conditional Probabilities
			Prompt-Based Generation
		6.5.3 Transferring a Document to Another Text Style
			Style Transfer with Parallel Data
			Style Transfer without Parallel Data
			Style Transfer with Few-Shot Prompts
		6.5.4 Story Generation with a Given Plot
			Specify a Storyline by Keywords or Phrases
			Specify a Storyline by Sentences
			Other Control Strategies
		6.5.5 Generating Fake News
			Detecting Fake News
		6.5.6 Generating Computer Code
			Available Implementations
		6.5.7 Summary
	6.6 Dialog Systems
		6.6.1 Dialog Models as a Pipeline of Modules
		6.6.2 Advanced Dialog Models
		6.6.3 LaMDA and BlenderBot 3 Using Retrieval and Filters
		6.6.4 Limitations and Remedies of Dialog Systems
			Available Implementations
		6.6.5 Summary
	References
7 Foundation Models for Speech, Images, Videos, and Control
	7.1 Speech Recognition and Generation
		7.1.1 Basics of Automatic Speech Recognition
		7.1.2 Transformer-Based Speech Recognition
		7.1.3 Self-supervised Learning for Speech Recognition
			Available Implementations
		7.1.4 Text-to-Speech
			Available Implementations
		7.1.5 Speech-to-Speech Language Model
		7.1.6 Music Generation
			Available Implementations
		7.1.7 Summary
	7.2 Image Processing and Generation
		7.2.1 Basics of Image Processing
		7.2.2 Vision Transformer
		7.2.3 Image Generation
		7.2.4 Joint Processing of Text and Images
		7.2.5 Describing Images by Text
		7.2.6 Generating Images from Text
		7.2.7 Diffusion Models Restore an Image Destructed by Noise
		7.2.8 Multipurpose Models
			Available Implementations
		7.2.9 Summary
	7.3 Video Interpretation and Generation
		7.3.1 Basics of Video Processing
		7.3.2 Video Captioning
		7.3.3 Action Recognition in Videos
		7.3.4 Generating Videos from Text
			Available Implementations
		7.3.5 Summary
	7.4 Controlling Dynamic Systems
		7.4.1 The Decision Transformer
		7.4.2 The GATO Model for Text, Images and Control
			Available Implementations
		7.4.3 Summary
	7.5 Interpretation of DNA and Protein Sequences
		7.5.1 Summary
	References
8 Summary and Outlook
	8.1 Foundation Models Are a New Paradigm
		8.1.1 Pre-trained Language Models
		8.1.2 Jointly Processing Different Modalities by Foundation Models
		8.1.3 Performance Level of Foundation Models
			Capturing Knowledge Covered by Large Text Collections
			Information Extraction
			Text Processing and Text Generation
			Multimedia Processing
		8.1.4 Promising Economic Solutions
	8.2 Potential Harm from Foundation Models
		8.2.1 Unintentionally Generate Biased or False Statements
			Accidentally Generated False or Misleading Information
			Reducing Bias by Retrieval
			Filtering Biased Text
		8.2.2 Intentional Harm Caused by Foundation Models
			Fake Images Created by Foundation Models
			Surveillance and Censorship
		8.2.3 Overreliance or Treating a Foundation Model as Human
		8.2.4 Disclosure of Private Information
		8.2.5 Society, Access, and Environmental Harms
			Access to Foundation Models
			Energy Consumption of Foundation Models
			Foundation Models Can Cause Unemployment and Social Inequality
			Foundation Models Can Promote a Uniform World View and Culture
			A Legal Regulation of Foundation Models Is Necessary
	8.3 Advanced Artificial Intelligence Systems
		8.3.1 Can Foundation Models Generate Innovative Content?
		8.3.2 Grounding Language in the World
		8.3.3 Fast and Slow Thinking
		8.3.4 Planning Strategies
	References
Appendix A
	A.1 Sources and Copyright of Images Used in Graphics
Index