Approximate dynamic programming for optimal stationary control with control-dependent noise

Area of Science:

• Control Theory
• Machine Learning
• Stochastic Systems

Background:

• Stochastic optimal control is crucial for systems with uncertainty.
• Reinforcement learning and adaptive dynamic programming (ADP) offer powerful tools for control.
• Existing methods may struggle with complex noise characteristics.

Purpose of the Study:

• To develop a reinforcement learning-based policy iteration algorithm for stochastic optimal control.
• To address problems with both additive and multiplicative noise using Itô calculus.
• To demonstrate the convergence properties and efficiency of the proposed ADP methodology.

Main Methods:

• Policy iteration algorithm derived using Itô calculus.
• Approximate/adaptive dynamic programming (ADP) framework.
• Analysis of convergence for the approximated cost matrix expectation.

Main Results:

• The expectation of the approximated cost matrix converges to the solution of an algebraic Riccati equation.
• The optimal cost value is determined by the algebraic Riccati equation.
• Reducing the covariance of the approximated cost matrix is achievable by adjusting iteration time intervals.

Conclusions:

• The proposed ADP methodology provides an efficient solution for stochastic optimal control.
• The algorithm guarantees convergence of the cost matrix expectation.
• Numerical examples validate the effectiveness of the adaptive dynamic programming approach.