LINMA2725: Stochastic optimal control and reinforcement learning

This is the public website for the Université catholique de Louvain course LINMA2725 Stochastic optimal control and Reinforcement Learning. This page is unofficial and for informational purposes only. Enrolled students should consult the course website hosted on Moodle.

Course description

This course covers foudations of stochastic optimal control, which concerns making optimal decisions in uncertain environments. Particular focus is given to Reinforcement Learning (RL) algorithms, which seek to make decisions without a model for the underlying system, but rather learn optimal policies online by observing rewards. Applications are investigated in problems in financial mathematics.

Course content

The corse covers:

  • Foundations of probabilities, systems, and optimal control

  • Algorithms for decision-making in deterministic environments

  • Algorithms for decision-making in stochastic environments

  • Control in discrete state-spaces and Markov decision processes

Learning outcomes

At the end of the course, the student will be able to:

  • Understand the concept of optimizing a stochastic process or system

  • Reformulate practical problems, especially from Financial Mathematics, as mathematical decision problems for stochastic systems

  • Utilize the foundational tools from stochastic optimal control to solve the decision problems

  • Apply algorithmic tools for the exact or approximate solving of optimal control problems, as well as understand their strengths and limitations and scope of applicability;

  • Understand the concept of exploitation vs exploration and regret minimization

  • Provide an exact or approximate solution to stochastic optimal decision problems

Project

The course includes a project to be realized in groups.

Tentative schedule

Week Description
1 Foundations of probabilities, system, and optimal control
2 LQR, LQG
3 Model Predictive Control
4 TD-learning and Linear Regression
5 Projected Bellman Equations, Convex Q-learning
6 Approximate Value-function and Q-function
7 Exploration vs exploitation
8 Regret minimization
9 Stochastic Gradient Descent & actor-critic

Textbooks

  • S. Meyn. Control systems and reinforcement learning. Cambridge University Press, 2022.

  • D. Bertsekas. Reinforcement learning and optimal control. Athena Scientific, 2019.