Yuejie Chi

S&DS 6190: Sequential Decision Making: Theoretical Foundations and Modern Applications

Sequential decision making in the face of uncertainty has garnered growing interest in recent years, with successes in a multitude of applications such as recommendation systems, robotics, and AI systems. This course aims to cover important theoretical and algorithmic foundations under the paradigms of bandits and reinforcement learning (RL), and discuss research issues arising from their modern applications in generative AI. We will cover multiple important topics including stochastic bandits, adversarial bandits, planning in Markov decision processes, online and offline RL, policy optimization, and multi-agent RL, gravitating our discussions around issues such as sample complexity, computational efficiency, and function approximation. We will also illustrate how RL is used to enable reasoning and alignment of foundation models, along with related (open) research questions.

Course Syllabus

Recommended Readings

• Reinforcement Learning: Theory and Algorithms, by Agarwal, Brantley, Jiang, Kakade, and Sun

• Reinforcement learning: An introduction, by Sutton and Barto

• Bandit Algorithms, by Lattimore and Szepesvari

• Reinforcement Learning, lecture notes by Silver

• Sample Complexity of Reinforcement Learning: A Non-Asymptotic Perspective, by Chen, Chi, Fan, Li, Wei, and Yan

Course Schedule

This is a provisional schedule and may be updated as the semester proceeds.