Dear students,
let me inform you that there will be no "lecture" on the upcoming Wednesday, March 16. I will be out of city for the whole week. But the afternoon exercises with Martin Gurtner will normally take place as usual.
Concerning the content for this upcoming 5th week, a usual dose of material from me (videos, lecture notes, quizzes, ...) is available on the course website.
But let me inform you that I have prepared this topic mostly using the book by Frank Lewis that I mentioned at the end of our lecture this week. And the nice thing is that Frank (I really call him like this because our dept has quite a tight relationship with him, he even visited us some time ago and his former phd student is with us) made it freely available on his website: https://lewisgroup.uta.edu/FL%20books/Lewis%20optimal%20control%203rd%20edition%202012.pdf. Please get the book, you may like it. The material for this week is covered in sections 6.1 and 6.2. Due to time constraints I typically skip 6.3, which extends the dynamic programming to the continuous time case, but have a look at it too, if you like, it is quite fundamental too, introducing the celebrated HJB equation (I also cover it on just a single page in my lecture notes).
Now, let me finally make a general comment on how to view dynamic programming. It is both a theoretical concept that can be used as yet another approach to deriving solutions to problems like LQ optimal control, and an algorithmic framework for solving general nonlinear optimal control problems. In this latter use, DP does not scale nicely – it suffers from the curse of dimensionality. Therefore some approximations are developed. The most prominent one is the framework of reinforcement learning. We are not going to cover it in this course, but mastering the very basics of DP will certainly make it easier for you to study RL on your own. Just a minor warning – reinforcement learning is one of those fields, which - having been simultaneously developed by several communities (control theory and computer science) – suffers from frequent terminology and notation clashes. In this regards, Chapter 10 in the book by Frank Lewis can serve as the least conflicting intro. I also like the relatively recent Reinforcement learning book by Dimitri Bertsekas. But once again, all this is well beyond our plans for this course. I will happily discuss it with those of you who venture out in this direction though.