Knowledge-augmented Methods for Natural Language Processing

Preface
Contents
1 Introduction to Knowledge-augmented NLP
	1.1 A Brief History of NLP
	1.2 What Large Language Model is Not Good At
	1.3 Common Knowledge Sources
	1.4 How to Integrate Knowledge into Language Model
	References
2 Knowledge Sources
	2.1 Introduction
	2.2 Pretrained Language Models
	2.3 Unstructured Knowledge
	2.4 Structured Knowledge
	2.5 Task Specific Corpus and Tool
	2.6 Conclusion
	References
3 Knowledge-augmented Methods for NaturalLanguage Understanding
	3.1 Introduction
	3.2 Tasks and Benchmarks
		3.2.1 Question Answering
		3.2.2 Knowledge-Graph Related Tasks
		3.2.3 Knowledge-Related Tasks in Other Applications
	3.3 Models and Methods for Structured Knowledge
		3.3.1 Knowledge Representation
		3.3.2 Knowledge Grounding
		3.3.3 Knowledge Fusion
	3.4 Models and Methods for Unstructured Knowledge
		3.4.1 Knowledge Representation
		3.4.2 Knowledge Grounding
		3.4.3 Knowledge Fusion
	3.5 Representative Models
		3.5.1 ERNIE
		3.5.2 Entity as Experts
		3.5.3 Dict-BERT
		3.5.4 Dense Passage Retrieval
		3.5.5 REALM
		3.5.6 REINA
		3.5.7 KEAR
		3.5.8 REPLUG
	3.6 Conclusion and Future Trends
	References
4 Knowledge-augmented Methods for Natural Language Generation
	4.1 Introduction
		4.1.1 What is Natural Language Generation?
		4.1.2 Why Knowledge Is Needed in Language Generation?
		4.1.3 What Is Knowledge-enhanced NLG?
	4.2 Incorporating Knowledge into NLG Models
		4.2.1 Knowledge Acquisition
			4.2.1.1 Sparse/Dense Retrieval
			4.2.1.2 Entity Linking
			4.2.1.3 Key-Value Retrieval
			4.2.1.4 Tool Utilization
		4.2.2 Knowledge Representation
			4.2.2.1 Knowledge as Condition
			4.2.2.2 Knowledge as Regularization
			4.2.2.3 Knowledge as Implicit Bias
			4.2.2.4 Knowledge as Feedback
	4.3 Representative Methods
		4.3.1 NLG Methods Enhanced by Text Retrieval
			4.3.1.1 REALM
			4.3.1.2 RAG
			4.3.1.3 FiD
			4.3.1.4 RETRO
			4.3.1.5 REPLUG
	References
5 Augmenting NLP Models with Commonsense Knowledge
	5.1 Commonsense Knowledge and Reasoning for NLP
		5.1.1 The Importance of Common Sense in NLP
		5.1.2 Knowledge Resources
		5.1.3 Reasoning Tasks
	5.2 Augmentation with Structured Knowledge
		5.2.1 Knowledge-Aware Graph Networks Fused with Language Models
			5.2.1.1 Overview of the KagNet Framework
			5.2.1.2 Schema Graph Grounding
			5.2.1.3 KagNet Network Design
		5.2.2 Multi-Hop Graph Relational Networks for Knowledge Fusion
		5.2.3 Advanced Graph-Language Fusion Methods
		5.2.4 Incorporating Structured Commonsense Knowledge for NLG
	5.3 Augmentation with Un/Semi-structuredKnowledge
		5.3.1 Open-Ended Reasoning with Commonsense Knowledge
		5.3.2 Dense Passage Retrieval for Open-Ended QA
		5.3.3 Differentiable Reasoning with Semi-Structured Knowledge
	5.4 Augmentation with Neural Knowledge Models
		5.4.1 Neural Knowledge Models of Common Sense
		5.4.2 LLMs as Knowledge Models
	5.5 Conclusion
		5.5.1 Summary
		5.5.2 Future Directions
	References
6 Summary and Future Directions
	6.1 Summary
		6.1.1 Knowledge-augmented NLU
		6.1.2 Knowledge-augmented NLG
		6.1.3 Knowledge-augmented Commonsense Reasoning
	6.2 Challenges and Future Opportunities
		6.2.1 Augmenting with Heterogeneous Knowledge
		6.2.2 Augmenting with Knowledge Beyond Passages
		6.2.3 Scaling Knowledge Augmentation
		6.2.4 Knowledge Augmentation for Structured Data Tasks
		6.2.5 Knowledge Augmentation for Faithfulness
		6.2.6 Knowledge Augmentation for NLG Diversity