M2 · Course resources
Risk-aware Reinforcement Learning
Risk-aware Reinforcement Learning - Master 2 Informatique : Concepts et Applications
Presentation slide
From September 11th to November 13th
The Hands-on homeworks will count 50% in total of the final note.
Course description
Beyond the classical theory, this course will cover some recent extensions of the theory of Markov Decision Processes (MDP) to the optimization of various risk measures. Starting from Bellman's Dynamic Programming, we will introduce the Distributional approach. We will then discuss how risk measures are usually defined, and discuss their properties. We will then show how classical algorithms based on backward induction can, or cannot, be adapted. The theory will be illustrated on classical or less classical benchmarks such as for example labyrinths or the inventory management problem. We will finally consider the foundations of reinforcement learning, starting with the vanilla bandit problem and then considering general approaches for MDPs. The course will be balanced between theory and practice, with exercice sessions dedicated to the implementation of the course’s algorithmic content.Presentation slide
Weekly schedule 2025-2026
Monday 10:15-12:15 and Thursday 10:15-12:15From September 11th to November 13th
Lecture notes
- Chapter 1: Sequential Decisions and Markov Decision Processes
- Chapter 2: Monte Carlo and the Statistics of Returns
- Chapter 3: Finite Horizon: the Distributional Bellman Equation
- Chapter 4: The Distributional Viewpoint and Risk
- Chapter 5: Utilities, Preferences and Distributional Planning Bellman Equation
- Chapter 6: Risk Measures: from Chernoff to EVaR and Risk
- Chapter 7: Discounted MDPs: Operators and Contractions and Risk
- Chapter 8: Optimal Planning in Discounted MDPs
- Chapter 13: Large State Spaces and Approximate DP (slides)
- Chapter 14: Policy Gradient (slides)
Hands-On Sessions
-
Notebook 1: The Bakery
Here is a baseline submission to be evaluated in various scenarios (it uses common.py for policy evaluation) - here is one
=> please upload your code here before Monday September 15th -
Notebook 2: The Photo Booth
=> please upload your code here before Monday September 29th -
Notebook 3: Windy Hike Around The Pit
=> please upload your code here before October 8th -
Notebook 4: Retail Store Management
=> please upload your code here before October 22nd -
Notebook 5: Machine Replacement
=> please upload your code here before November 5th -
Notebook 6: Hike with Function Approximation
(it uses common2.py for policy evaluation)
=> please upload your code here before November 12th
The Hands-on homeworks will count 50% in total of the final note.
Prerequisite
Notions of Machine Learning, elementary probability and statistics, elementary linear algebra.
Bibliography
- Algorithms for Reinforcement Learning, by Csaba Szepesvari (2010)
- Reinforcement Learning: Theory and Algorithms, by Alekh Agarwal Nan Jiang Sham M. Kakade Wen Sun (2024)
- Bandit Algorithms, by Tor Lattimore and Csaba Szepesvári (2016)
- Markov Decision Processes: Discrete Stochastic Dynamic Programming, by Martin L. Puterman (2005)
- Reinforcement Learning: An Introduction, by Richard S. Sutton and Andrew G. Barto (2018)
- Dynamic Programming and Optimal Control, by Dimitri Bertsekas
- Deep Reinforcement Learning, by Aske Plaat
- Distributional Reinforcement Learning, By Marc Bellemare, Will Dabney and Mark Rowland
