Dear students, although we have discussed this issue with some of you during the exercises, let me clarify it here once again:
Indeed, maybe it is not clear from the assignment of the HW5, but the proposed discretization of the state variable x and the control variable u should only be used while building those lookup tables J*(x,k) and u*(x,k). But during simulation, both the state and the control variable should be regarded continuous. Maybe it was hinted by "Use linear interpolation in cases where you need to get a value of the value function 𝐽∗ or optimal control policy 𝑢∗ outside the quantized values of 𝑥", but perhaps it should have been more explicit.
If this clarification comes too late for you, and you have either submitted the solution or have no time to incorporate it, do not worry. I do admit that it was not perfectly clear and I will take it into consideration during evaluation.