Assigned (compulsory) reading

No compulsory reading.

Recommended (not compulsory) further reading

Our introductory treatment of dynamic programming is to a large extent based on Chapter 6 in [1]. Note however that newer edition of the book is now on the market (and even freely downloadable from the first author's webpage).

Some other classics that are perhaps better accessible (because cheaper) such as [2] also contain comparable introductory exposition. Another classic [3] actually uses dynamic programming as the only "vehicle" to derive all those LQ-optimal regulation and tracking results. A few copies of this book are available in the faculty library at NTK. Fairly detailed treatment is in the two-volume [4]

In our introductory lecture we mainly regarded dynamic programming as a theoretical concept which enabled us to (re)derive some analytical results such as discrete-time and continuous-time LQ-optimal control. The usefulness of dynamic programming as a practical computational scheme is fairly limited because of the curse of dimensionality problem (the computational complexity and memory requirements grow quickly with the dimension of the state space). These deficiencies of dynamic programming are attacked by methodologies known under various names as neuro-dynamic programming, approximate dynamic programming or reinforcement learning. We will not cover these in our course, but an interested reader can find an accessible introduction in [5] (in fact the same material as the chapter 11 of the latest edition of [1] available online). As a bible of reinforcement learning, [6] is often mentioned (available for free download). But the fact is that it is written from a computer science perspective, which might make its reading a bit difficult for a control engineer. Two recent monographs are particularly recommendable as introductions to reinforcement learning from control systems perspective –[7] and [8], both having free drafts available online.

[1] Frank L. Lewis and Vassilis L. Syrmos. Optimal Control, 2nd Edition. Wiley-Interscience, October 1995. The 3rd edition from 2012 available for free download on the author's webpage.

[2] D. E. Kirk, Optimal Control Theory: An Introduction. Dover Publications, 2004.

[3] B. D. O. Anderson and J. B. Moore, Optimal Control: Linear Quadratic Methods. Dover Publications, 2007.

[4] D. P. Bertsekas. Dynamic Programming and Optimal Control. Vol. I and II. Athena Scientific. 2017.

[5] F. L. Lewis, D. Vrabie, K. Vamvoudakis. Reinforcement Learning and Feedback Control: Using Natural Decision Methods to Design Optimal Adaptive Controllers. IEEE Control Systems, December 2012. https://doi.org/10.1109/MCS.2012.2214134.

[6] R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, Second edition. Cambridge, Massachusetts: A Bradford Book, 2018. [Online]. Available: http://incompleteideas.net/book/the-book-2nd.html.
 
[7] D. Bertsekas, Reinforcement Learning and Optimal Control, 1st edition. Athena Scientific, 2019. [Online]. Available: https://www.mit.edu/~dimitrib/RLbook.html.

[8] S. Meyn, Control Systems and Reinforcement Learning. Cambridge University Press, 2022. Accessed: Aug. 25, 2021. [Online]. Available: https://meyn.ece.ufl.edu/control-systems-and-reinforcement-learning/

Last modified: Wednesday, 22 March 2023, 2:05 PM