IoT System Testing: An IoT Journey from Devices to Analytics and the Edge

1
	Contents
	About the Author
	About the Technical Reviewer
	Acknowledgments
2
	Part 1: Getting Started
978-1-4842-8276-2.webp
978-1-4842-8276-2_1
	Chapter 1: The Internet of Things, V&V, and Testing
		IoT at a Glance
		Understanding Chaos, Security, and Other Product Qualities
		Why Test IoT Beyond the Device Itself
		Agile vs. Traditional vs. “Who Cares About It” Development
		This Book’s Audience
		How to Use This Book
			Reference Standards, Books, and the Internet: Context Matters
			Valuable References to Start Your IoT Test Library
		Summary
		References
			Figure Reference
978-1-4842-8276-2_2
	Chapter 2: IoT Technology in Time and Space
		IoT at a Glance
		IoT Market Segments – Where Is IoT Now and in the Future?
		A Sampling of IoT Challenges in Development and Testing
		IoT Test Team Responses to the Development History of “Testing Is Dead”
			Examples of How the IoT Devices Can Impact Testers
		Summary
		References
			Figure References
978-1-4842-8276-2_3
	Chapter 3: Big Picture Lessons Learned in IoT Project Test Planning
		High-Level Issues at a Glance
			#1: Creating and Releasing Software
			#2: Understanding the IoT Lifecycle
			#3: Test Tools for IoT
			#4: Avoid Impacts of Rushing in Competition
			#5: Advanced IoT Device Challenges
			#6: Testing in the Complex World of IoT Systems and Systems of Systems
		Project Test Risks: What Are They for IoT?
		Getting Started with IoT Test Plans and Strategy
			Introducing IoT Verification, Validation, and Testing Concepts and Standards
				What Is IoT V&V in a Nutshell
		Summary
		References
			Figure Reference
978-1-4842-8276-2_4
	Chapter 4: Factors Driving IoT Testing/V&V Selection and Planning
		Factor 1: Using Integrity Levels to Drive V&V/Test Planning and Strategy
		Factor 2: Risk-Based DevOps and Product Maturity Testing
			Risk Analysis Process for Testers
		Factor 3: Organizational Ability Impacts on IoT Test Planning
			“Newbie” Companies – Level 1
			Companies with Experience Moving into IoT – Level 2
			Companies with Hardware (Electronics) Experience – Level 3
			Companies with Software Experience – Level 3
			Company with Hardware-Software Experience – Level 4
			Companies with Systems, Hardware, and Software Experience – Level 5
			Government Organization – No Level but a Special Case
			Experienced Companies with Only Consumer Product History – Level 2
				Highly Experienced Company Including an IoT History – Level 5+
		Factor 4: IoT Project Size and Complexity Impacts on Testing
			Composite Total Scoring Factors (If You Need to Do That)
			How IoT Projects Should Mix and Match Factors
			Combining IoT Factors for Better Test Planning
			IoT Test/V&V Cost Estimation
		Summary
		References
			Figure References
978-1-4842-8276-2_5
	Chapter 5: Beginner Keys for Starting IoT Test Planning
		IoT Key 1: Have a Ubiquitous User Interface (UI)
		IoT Key 2: Learning from Data Analytics
		IoT Key 3: Unique and Specialized Hardware Working to Be a System with the Software
		IoT Key 4: Level of V&V/Test Need for “Good Enough” IoT
		IoT Key 5: Remaining Agile
		IoT Key 6: Testing IoT, Systems, and Large/Complex Software (LCS)
		Summary
		References
978-1-4842-8276-2_6
	Chapter 6: IoT Test Plan: Strategy and Architecture Introductions
		Establishing the Mindset
			Additional Considerations
		Planning
			Good Enough IoT Software and Devices
		IoT Test Planning Basics
		General Test Planning Outlined by Organization Classification
			The Pure Startup, Single Device, and Small Team Who Are Trying to Stay Alive
			Mature Groups or Growing Teams Targeting Growth of Sales
			IoT Test for a System with More Devices
			IoT Test Planning for Procuring Organizations: Governments, Large Corporations, and Others
		Impact of AI, Data, and Analytics on IoT Test Planning
			Product and Development Lifecycle Impacts on Test Planning – DevOps and Agile
		Summary
		References
			Figure Reference
978-1-4842-8276-2_7
	Chapter 7: IoT Test Planning and Strategy for Hardware and Software
		Traditional Testing Overview
		Planning in the Agile/DevOps IoT Test Lifecycle
			What Should Development and Test Do During Agile and DevOps?
			Ops Team People Skilled in Agile Test Thinking
			Planning Using a Hybrid Agile Test Lifecycle
		IoT Test Plan Budgeting, Estimating, and Scheduling
			Estimating Schedules – A Brief Introduction to Support Cost Estimates
			Estimation of Testing Size
			Quality, Verification, Validation, and Testing
			Test and Quality
			Verification and Validation Activity
				Wrap Up Test Planning: Regression Test Cases in IoT
		IoT Test Planning: OTS Hardware and Software
			Management Activities of the Coil
			Engineering Using the Coil
			Software Dev-Test-Sec-Ops Team in the Coil
		IoT Test Planning: Security and Critical Quality Factors
			IoT Test Planning: Introduction to Software Test Architecture (STA)
		IoT Test Planning: IoT Strategy
			Strategy – The Basics
			IoT Test Strategy
			What Is a Checklist for Strategy Selection?
			What Is a Strategy for the Individual IoT Tester?
			IoT System V&V Planning: Start with a Combination of Test Strategies
			Hardware V&V Strategy (Chapters 8, 12, and 19)
			Software V&V Strategy
			Ops V&V Strategy
		Hardware Test/V&V Planning (IEEE 1012)
		Test Planning for Software Systems
		IoT System Integration Test Planning
		IoT System and System of Systems V&V/Test Planning – A Conceptual Overview
		Summary
		References
			Figure Reference
978-1-4842-8276-2_8
	Chapter 8: Planning for the IoT Tester on Environments and Testing Details
		Assuring the Test Environment
			Selecting the Right Test Environment
			Planning for Automation in IoT Tool Environments
			Data Analytics with Tools
		IoT Detailed Project Test Planning (After the Master Test Plans)
			Hardware Planning a Tester Should Know
			Detailed Software Testing
			System Test Plan Patterns a Tester Should Know
			Planning Individual Tests (What All Testers Should Do Daily)
		Test Planning from Operations to the End Product Life
			Test Operations (Ops) Impacts on Test Planning
			How Does Ops and Test Planning Change Over Time?
			System Maintenance, Security, and Retirement
			Ops Maintenance in Test Planning
			Planning Retirement and Disposal of an IoT System
			Testing Integration Factors
			Planning Test IoT Operations and Maintenance (O&M) with Data Analytics and the Edge
			Test Planning for Release Deployment
			Factory Production: A Very Brief Introduction to Testing IoT Hardware
				Test and Quality
				Evolving As You Go
				Minimum Activities Needed to Release an IoT Device into the Wild First Pass
			Risk and Opportunity Management on an IoT Test Project
			Last but Not Least, Test Documentation
		Summary
		References
978-1-4842-8276-2_9
	Chapter 9: System Engineering Concepts in IoT Test Planning
		Reviewing Basic Software Engineering (SE) Concepts
		Critical Test Enabling Engineering Support Processes
			SQA/QA
			SCM and Testing
				Software (Test) Measurements
				Measurement and Status Supporting Systems Engineering and Testing
			System Architecture and Design – Test Top-Level Support
			System Planning Trade Study – Decision Analysis
		Designing with Safety for IoT
		Hardware Design Considerations
		Software Design Considerations
		IoT System Integration
		Agile and DevOps Development Impacts
		Summary
		References
			Figure Reference
978-1-4842-8276-2_10
	Chapter 10: IoT Test Design: Frameworks, Techniques, Attacks, Patterns, and Tours
		Test and Heuristics
		Test Patterns
		Example 1: Planning Pattern for IoT [8, 9]
		Example 2: Mind Maps – Test Patterning Tool for Process Selection for IoT
		Example 3: Attacks for IoT – High-Level Test Design Pattern
		Example 4: Test Meta-design Pattern – Tours for IoT
			Applying the Specific Tours to Project Factors
		Example 5: Frameworks – The Top Level of Test Design Planning
		Summary
		References
978-1-4842-8276-2_11
	Chapter 11: Classic IoT V&V/Test Concepts, Techniques, and Practices
		Moving from Simple Testing to Providing Information on Quality
		Techniques, Practices, Levels, and Types of Testing to Apply to IoT
		Functional Test Design Techniques
			Requirements Based
			Boundary Value Analysis (BVA)
			Scenario (a.k.a. Use Case and Stories) Testing
			Syntax of IoT Command Structure Testing
		Exploratory Testing
		Structural Testing
		Industrial Test Practices
			Math-Based Testing for IoT System and Software
			Model-Based Development and Testing/V&V – A Highly Automated and Integrated Test Practice
		IoT Levels of Testing Related to Lifecycle Phases
		Summary
		References
978-1-4842-8276-2_12
	Chapter 12: Test Approaches and Quality Assessments for IoT Agile/DevOps
		Working Without a Formal IoT Test Plan (or Any Other Plans)
		Test Planning for the Agile Teams
		Agile Grows and Evolves (and Will Continue to Change)
		Agile Hardware
		Agile Security Testing
		Agile Quality V&V Assessment
		Summary
		References
			Figure References
978-1-4842-8276-2_13
	Chapter 13: IoT Software Security Test Attacks and Designs
		Before We Get Started – Security Data Points and Warnings
			Definitions Used in This Chapter
		Using Attack Surfaces in Security Test Planning
		Planning Security Tests: IoT System Level Inspection and Assessment
		Key Activity: Risk and Vulnerability Analysis
			IoT Security Threat Risks and How to Find Them
		Cybersecurity Team: Test/V&V Supports Developer-Operations (DevOps) Security
			Key Activity: Cyber Kill Chain Model
			Key Activity: Using a Zero-Trust Security Architecture Model to Support IoT Testing
			Key Activity: Software Security Testing Cycle
				Social Engineering
				Penetration (Pen) Testing
				Access Sensitive Information
				Access Control and Password Cracking
				Elevate Access Privileges
				Denial of Service (DoS) Attack
				Tampering Attack
				Malware Attacks
				Security Attacking the Off-the-Shelf (OTS) Software for Malware
					Using a Table for Detection of Malware in the IoT Off-the-Shelf (OTS) Software
				Security Taint Analysis
				Spoofing [11]
				Edge, Fog, and Cloud Attack: Security Testing Needed beyond a Single IoT Device
		Historic Security Books You Should Have and Know
		Security Test Tool Categories
			Tools for Software Security Testing
			IoT Testing Tools for Hardware
			Software-Defined Security and AI – An IoT Future
			Security Finding Reporting
		Summary
		References
978-1-4842-8276-2_14
	Chapter 14: Security OWASP IoT Information Pointer and Logging Events
		Intro to OWASP Top Ten Threats (As of 2022)
		Summary
978-1-4842-8276-2_15
	Chapter 15: Internal Security Team Penetration Test Process
		Pen Test Process: A Beginning
			Why Perform Pen Test?
		Pen Security Attack and Risk-Based Test Planning for Systems
		Full-Scale Penetration of the System
		Simplified Process to Perform a Pen Test
		Summary
		References
978-1-4842-8276-2_16
	Chapter 16: IoT Test Environment Introduction
		Test Lab Lifecycle
		Test Lab Refresher
		IoT Lab/SIL Planning and Requirements
		Preferred Test Environment with Full Integration of a Complex IoT System
		The Field Test Environment, Analytics, and SIL Working Together
		Deep Dive on Test Environment Cost and Schedule Introduction
		Test Tool Introduction for IoT
		Test Hardware Setup for SIL-Chaos Engineering Support with ZIF Connectors
		Summary
		References
			Figure Reference
978-1-4842-8276-2_17
	Chapter 17: Architectures Critical to Project Success
		Cyber-Physical Systems
		Environments and Architectures for IoT
		Architecture Definition of Terms for This Book
		Historical Reference: Architecture in Engineering and Literature
		Summary
		References
978-1-4842-8276-2_18
	Chapter 18: Overview of IoT Software Architectures: Products and Testing Support
		A Quick Look at IoT Architectures
		Overview of IoT Support Architectures
		Mind Maps of IoT Environments
		IoT Software Test Architecture (STA) Introduction
		IoT Software Test Architecture (STA) Details
			Major Element: Test Plan (a.k.a. STAp)
			Major Element: System/Software Under Test (SUT)
			Major Element: Test Environment (STAe)
			Major Element: Test Model (STAm)
			Major Architecture Element: Views, Viewpoints, and Containers (STAv)
		Summary
		References
978-1-4842-8276-2_19
	Chapter 19: IoT STA System: Software Integration Lab (SIL) Environments
		Environment: Development Team Testing and Integration Support
		Environment: Hardware Team Testing and Integration
		Environment: Full Hardware-Software-System Integration
			A Simple Integration STA SIL
			A More Advanced IoT STA SIL with Rapid Integration Reconfiguration
		Test Environment Supports: Simulation, Modeling, and Emulation
			Full System IoT STA SIL with Simulation and Modeling
		Environment: Real-World Full System Software Test
		Special Environment: Security Test Sandbox
			IoT Chaos Engineering “Live” in the Real World
			Level of SIL Environment vs. Project Factors
			Bringing STA to a Large-Scale Software Test Architecture/Environment
		Summary
		References
			Figure Reference
978-1-4842-8276-2_20
	Chapter 20: Tools for the Software System Integration Lab (SIL)
		Test Lab Needs
		Modeling and Requirements Management Tools
		Automate, Automate, Automate
		Simulation Test Tools Needed to Support IoT
			Evolve into Automation Tooling for IoT Success
			Evolve IoT Testing with AI Tools
		Summary
		References
978-1-4842-8276-2_21
	Chapter 21: Environments for Independent Testing and IV&V on Large IoT Systems
		Getting the Most Out of Independent Testing and IV&V
			Verification
			Validation
		N-Version Testing Supporting Independence and IV&V
		Lessons Learned in Testing, in Independence, and IV&V
		Tooling Example for Testing Large IoT Systems and IV&V
		When to Consider IoT Test Independence and IV&V Environments
		Summary
		References
978-1-4842-8276-2_22
	Chapter 22: Self-Organizing Data Analytics (SODA): IoT Data Analytics, AI, and Statistics
		SODA Model Examined
		Defining SODA
		SODA Implementation Options Using Stats, Taxonomies, and AI
		Use Case Example for SODA and Edge Data Flow
		Leveraging SODA for IoT Testing
		AI SODA – A Near Future for IoT/Edge/Cloud Data Analytics
		Summary
		References
978-1-4842-8276-2_23
	IoT Supporting Interface, Hardware, Platform, and Protocol Standards
	Careers in IoT Testing
	IoT Testing Startup Checklist
	Getting Started with IoT Testing
		Usability Testing
	Compatibility and Integration Testing
	App Localization Issue
	External Connectivity Issues
	Interrupt Testing
	Operational Testing
	Installation Test
	Miscellaneous
	Example of an IoT System of Systems
	Index