摘要&结论

证明了VI框架下infinite horizon discrete-time (DT) nonlinear optimal control HJB方程（Hamilton–Jacobi–Bellman equation）的收敛性
解最优控制的方法:actor-critic方法

a critic NN is used to approximate the value function

**an action network **is used to approximate the optimal control policy

优点：不需要知道systems dynamic的完整知识

用来解决infinite horizon discrete-time (DT) nonlinear optimal control问题

特别的可以很好的用来解决LQR问题（DT linear quadratic regulator）

introduction

The DT nonlinear optimal control solution relies on solving the DT Hamilton–Jacobi– Bellman (HJB) equation

These are all offline methods for solving the HJB equations that require full knowledge of the system dynamics.

In this paper, we provide a full rigorous proof of convergence of the value-iteration-based HDP algorithm to solve the DT HJB equation of the optimal control problem for general nonlinear DT systems.

The point is stressed that these results also hold for the special LQR case

Section II starts by introducing the nonlinear DT optimal control problem.

Section III demonstrates how to set up the HDP algorithm to solve for the nonlinear DT optimal control problem.

In Section IV, we prove the convergence of HDP value iterations to the solution of the DT HJB equation.

In Section V, we introduce two NN parametric structures to approximate the optimal value function and policy.

II. DT HJB EQUATION

$V\left(x_{k}\right)=\sum_{n=k}^{\infty} Q\left(x_{n}\right)+u^{T}\left(x_{n}\right) R u\left(x_{n}\right)$

（根据某些条件）可以写成

$V\left(x_{k}\right)=x_{k}^{T} Q x_{k}+u_{k}^{T} R u_{k}+\sum_{n=k+1}^{\infty} x_{n}^{T} Q x_{n}+u_{n}^{T} R u_{n}$ $=x_{k}^{T} Q x_{k}+u_{k}^{T} R u_{k}+V\left(x_{k+1}\right)$

DT HJB equation：

$V^{*}\left(x_{k}\right)=\min _{u_{k}}\left(x_{k}^{T} Q x_{k}+u_{k}^{T} R | u_{k}+V^{*}\left(x_{k+1}\right)\right)$

$u^{*}$ 满足first-order necessary condition，所以对等式右边求梯度

$\frac{\partial\left(x_{k}^{T} Q x_{k}+u_{k}^{T} R u_{k}\right)}{\partial u_{k}}+\frac{\partial x_{k+1}}{\partial u_{k}} \frac{\partial V^{*}\left(x_{k+1}\right)}{\partial x_{k+1}}=0$

然后 $u^{*}$ 沿着梯度方向更新

$u^{*}\left(x_{k}\right)=\frac{1}{2} R^{-1} g\left(x_{k}\right)^{T} \frac{\partial V^{*}\left(x_{k+1}\right)}{\partial x_{k+1}}$

III. HDP ALGORITHM

见 [论文研读 Optimal and Autonomous Control Using Reinforcement Learning: A Survey]的 1) Offline PI Algorithm

V. NN APPROXIMATION FOR VALUE AND ACTION

论文研读 DT Nonlinear HJB Solution Using ADP: Convergence Proof

摘要&结论

introduction

II. DT HJB EQUATION

III. HDP ALGORITHM

V. NN APPROXIMATION FOR VALUE AND ACTION