Evaluating Information Retrieval and Access Tasks. NTCIR's Legacy of Research Impact

Foreword
Preface
Contents
Acronyms
1 Graded Relevance
	1.1 Introduction
	1.2 Graded Relevance Assessments, Binary Relevance Measures
		1.2.1 Early IR and CLIR Tasks (NTCIR-1 Through -5)
		1.2.2 Patent (NTCIR-3 Through-6)
		1.2.3 SpokenDoc/SpokenQuery& Doc  (NTCIR-9 Through -12)
		1.2.4 Math/MathIR (NTCIR-10 Through -12)
	1.3 Graded Relevance Assessments, Graded Relevance Measures
		1.3.1 Web (NTCIR-3 Through-5)
		1.3.2 CLIR (NTCIR-6)
		1.3.3 ACLIA IR4QA (NTCIR-7 and -8)
		1.3.4 GeoTime (NTCIR-8 and -9)
		1.3.5 CQA (NTCIR-8)
		1.3.6 INTENT/IMine (NTCIR-9 Through 12)
		1.3.7 RecipeSearch (NTCIR-11)
		1.3.8 Temporalia (NTCIR-11 and -12)
		1.3.9 STC (NTCIR-12 Through -14)
		1.3.10 WWW (NTCIR-13 and -14) and CENTRE (NTCIR-14)
		1.3.11 AKG (NTCIR-13)
		1.3.12 OpenLiveQ (NTCIR-13 and -14)
	1.4 Summary
	References
2 Experiments on Cross-Language Information Retrieval Using Comparable Corpora of Chinese, Japanese, and Korean Languages
	2.1 Introduction
	2.2 Outline of Cross-Language Information Retrieval (CLIR)
		2.2.1 CLIR Types and Techniques
		2.2.2 Word Sense Disambiguation for CLIR
		2.2.3 Language Resources for CLIR
	2.3 Test Collections for CLIR from NTCIR-1 to NTCIR-6
		2.3.1 Japanese-English Comparable Corpora in NTCIR-1 and NTCIR-2
		2.3.2 Chinese-Japanese-Korean (CJK) Corpora from NTCIR-3 to NTCIR-6
		2.3.3 CJKE Test Collection Construction
		2.3.4 IR System Evaluation
	2.4 CLIR Techniques in NTCIR
		2.4.1 Monolingual Information Retrieval Techniques
		2.4.2 Bilingual Information Retrieval (BLIR) Techniques
		2.4.3 Multilingual Information Retrieval (MLIR) Techniques
	2.5 Concluding Remarks
	References
3 Text Summarization Challenge:  An Evaluation Program for Text Summarization
	3.1 What is Text Summarization?
	3.2 Various Types of Summaries
	3.3 Evaluation Metrics for Text Summarization
	3.4 Text Summarization Evaluation Campaigns Before TSC
	3.5 TSC: Our Challenge
		3.5.1 TSC1
		3.5.2 TSC2
		3.5.3 TSC3
	3.6 Text Summarization Evaluation Campaigns After TSC
	3.7 Future Perspectives
	References
4 Challenges in Patent Information Retrieval
	4.1 Introduction
	4.2 Overview of NTCIR Tasks
		4.2.1 Technology Survey
		4.2.2 Invalidity Search
		4.2.3 Classification
		4.2.4 Mining
	4.3 Outline of the NTCIR Tasks
		4.3.1 Technology Survey Task: NTCIR-3
		4.3.2 Invalidity Search Task: NTCIR-4, NTCIR-5, and NTCIR-6
		4.3.3 Patent Classification Task: NTCIR-5, NTCIR-6
		4.3.4 Patent Mining Task: NTCIR-7, NTCIR-8
	4.4 Contributions
		4.4.1 Preliminary Workshop
		4.4.2 Technology Survey
		4.4.3 Collaboration with Patent Experts
		4.4.4 Invalidity Search
		4.4.5 Patent Classification
		4.4.6 Mining
		4.4.7 Workshops and Publications
		4.4.8 CLEF-IP and TREC-CHEM
	References
5 Multi-modal Summarization
	5.1 Background
	5.2 Applications Envisioned
	5.3 Multi-modal Summarization on Trend Information
		5.3.1 Objective
		5.3.2 Data Set as a Unifying Force
		5.3.3 Outcome
	5.4 Implication
	References
6 Opinion Analysis Corpora Across Languages
	6.1 Introduction
	6.2 NTCIR MOAT
		6.2.1 Overview
		6.2.2 Research Questions at NTCIR MOAT
		6.2.3 Subtasks
		6.2.4 Opinion Corpus Annotation Requirements
		6.2.5 Cross-Lingual Topic Analysis
	6.3 Opinion Analysis Research Since MOAT
		6.3.1 Research Using the NTCIR MOAT Test Collection
		6.3.2 Opinion Corpus in News
		6.3.3 Current Opinion Analysis Research: The Social Media Corpus and Deep NLP
	6.4 Conclusion
	References
7 Patent Translation
	7.1 Introduction
	7.2 Innovations at NTCIR
		7.2.1 Patent Translation Task at NTCIR-7 (2007–2008)
		7.2.2 Patent Translation Task at NTCIR-8 (2009–2010)
		7.2.3 Patent Translation Task at NTCIR-9 (2010–2011)
		7.2.4 Patent Translation Task at NTCIR-10 (2012–2013)
	7.3 Developments After NTCIR-10
	References
8 Component-Based Evaluation for Question Answering
	8.1 Introduction
		8.1.1 History of Component-Based Evaluation in QA
		8.1.2 Contributions of NTCIR
	8.2 Component-Based Evaluation in NTCIR
		8.2.1 Shared Data Schema and Tracks
		8.2.2 Shared Evaluation Metrics and Process
	8.3 Recent Developments in Component Evaluation
		8.3.1 Open Advancement of Question Answering
		8.3.2 Configuration Space Exploration (CSE)
		8.3.3 Component Evaluation for Biomedical QA
	8.4 Remaining Challenges and Future Directions
	8.5 Conclusion
	References
9 Temporal Information Access
	9.1 Introduction
	9.2 Temporal Information Retrieval
		9.2.1 NTCIR-8 GeoTime Task
		9.2.2 NTCIR-9 GeoTime Task Round 2
		9.2.3 Issues Discussed Related to Temporal IR
	9.3 Temporal Query Analysis and Search Result Diversification
		9.3.1 NTCIR-11 Temporal Information Access Task
		9.3.2 NTCIR-12 Temporal Information Access Task Round 2
		9.3.3 Implications from Temporalia
	9.4 Related Work and Broad Impacts
	9.5 Conclusion
	References
10 SogouQ: The First Large-Scale Test Collection with Click Streams Used in a Shared-Task Evaluation
	10.1 Introduction
	10.2 SogouQ and Related Data Collections
	10.3 SogouQ and NTCIR Tasks
	10.4 Impact of SogouQ
	10.5 Conclusion
	References
11 Evaluation of Information Access with Smartphones
	11.1 Introduction
	11.2 NTCIR Tasks for Information Access with Smartphones
		11.2.1 NTCIR 1CLICK
		11.2.2 NTCIR MobileClick
	11.3 Evaluation Methodology in NTCIR 1CLICK and MobileClick
		11.3.1 Textual Output Evaluation
		11.3.2 From Nuggets to iUnits
		11.3.3 S-Measure
		11.3.4 M-Measure
	11.4 Outcomes of NTCIR 1CLICK and MobileClick
		11.4.1 Results
		11.4.2 Impacts
	11.5 Summary
	References
12 Mathematical Information Retrieval
	12.1 Introduction
	12.2 NTCIR Math: Overview
		12.2.1 NTCIR-10 Math Pilot Task
		12.2.2 NTCIR-11 Math-2 Task
		12.2.3 NTCIR-12 MathIR Task
	12.3 NTCIR Math Datasets
		12.3.1 Corpora
		12.3.2 Topics
		12.3.3 Relevance Judgment
	12.4 Task Results and Discussion
		12.4.1 Evaluation Metrics
		12.4.2 MIR Systems
	12.5 Further Trials
		12.5.1 ArXiv Free-Form Query Search at NTCIR-10
		12.5.2 Wikipedia Formula Search at NTCIR-11
		12.5.3 Math Understanding Subtask at NTCIR-10
	12.6 Further Impact of NTCIR Math Tasks
		12.6.1 Math Information Retrieval
		12.6.2 Semantics Extraction in Mathematical Documents
		12.6.3 Corpora for Math Linguistics
	12.7 Conclusion
	References
13 Experiments in Lifelog Organisation and Retrieval at NTCIR
	13.1 Introduction
	13.2 Related Activities
	13.3 Lifelog Datasets Released at NTCIR
	13.4 Lifelog Subtasks at NTCIR
		13.4.1 Lifelog Semantic Access Subtask
		13.4.2 Lifelog Insight Subtask
		13.4.3 Other Subtasks (LEST, LAT and LADT)
	13.5 Lessons Learned
		13.5.1 Conclusions and Future Plans
	References
14 The Future of Information Retrieval Evaluation
	14.1 Introduction
	14.2 First Things First
	14.3 The Shared Task Evaluation Ecosystem
	14.4 A Brave New World
	14.5 Trendlines
	14.6 An Inconclusion
	References
Index