Reinforcement Learning for Reconfigurable Intelligent Surfaces

Preface
Contents
Acronyms
1 Reinforcement Learning Background
	1.1 Overview
	1.2 Discrete Spaces
		1.2.1 Q-Learning
		1.2.2 Deep Q-Learning
	1.3 Continuous Spaces
		1.3.1 DDPG
	References
2 RIS-Assisted Wireless Systems
	2.1 Overview
	2.2 Scenarios
		2.2.1 RIS-Assisted Cognitive Radio Networks
		2.2.2 RIS-Assisted Unmanned Aerial Vehicle
		2.2.3 RIS-Assisted Simultaneous Wireless Information and Power Transfer
	2.3 System Models
		2.3.1 Half-Duplex
		2.3.2 Full-Duplex
	References
3 Applications of RL for Continuous Problems in RIS-Assisted Communication Systems
	3.1 Application 1: Maximizing Sum Rate
		3.1.1 Action Space
		3.1.2 State Space
		3.1.3 Reward
	3.2 Application 2: Maximizing the Weighted Sum Rate
		3.2.1 Action Space
		3.2.2 State Space
		3.2.3 Reward
	3.3 Application 3: Maximizing the Location-Based Achievable Rate
		3.3.1 Action Space
		3.3.2 State Space
		3.3.3 Reward
	3.4 Application 4: Maximizing the Energy Efficiency
		3.4.1 Action Space
		3.4.2 State Space
		3.4.3 Reward
	3.5 Application 5: Maximizing the Secrecy Rate
		3.5.1 Action Space
		3.5.2 State Space
		3.5.3 Reward
	References
4 Applications of RL for Discrete Problems in RIS-Assisted Communication Systems
	4.1 Application 6: Maximizing Sum Rate
		4.1.1 Action Space
		4.1.2 State Space
		4.1.3 Reward
	4.2 Application 7: Minimizing System Resources
		4.2.1 Action Space
		4.2.2 State Space
		4.2.3 Reward
	4.3 Application 8: Maximizing the Energy Efficiency
		4.3.1 Action Space
		4.3.2 State Space
		4.3.3 Reward
	4.4 Application 9: Maximizing the Spectral Efficiency
		4.4.1 Action Space
		4.4.2 State Space
		4.4.3 Reward
	4.5 Application 10: Maximizing the Minimum User Spectral Efficiency
		4.5.1 Action Space
		4.5.2 State Space
		4.5.3 Reward
	References
5 Challenges and Future Work
	5.1 Challenges
		5.1.1 Hyperparameter Tuning and Problem Design
		5.1.2 Complexity Analysis
	5.2 Future Work
		5.2.1 Hybrid RL
			5.2.1.1 DRL and Supervised Learning
			5.2.1.2 Combined DRL Algorithms
		5.2.2 Exploiting Multi-Agent DRL
		5.2.3 Incorporating Transfer Learning into DRL
	5.3 Concluding Remarks
	References