Policy gradient usually requires integral over all the possible actions.


The purpose of reinforcement learning is to learn the policy to maximize the objective function. Policy gradient directly train the policy network to minimize the objective function.

  • Stochastic Policy Gradient
    Since this assumes stochastic policy, this is called Stochastic Policy Gradient. If a sample return is used to estimate the action-value function, it is called REINFORCE algorithm.

  • Stochastic Actor-Critic
    We can train another network to directly learn the value of action-value function by td learning.

  • Off-policy Actor-Critic (OffPAC)
    On-policy learning has limitation in exploration. Off-policy learning use different policies to behave and to evaluate. This Off-Policy Actor-Critic(OffPAC)require importance sampling.

  • Deterministic Policy Gradient
    In continuous action space, integral over all the action space is intractable. Deterministic policy gradient uses the deterministic policy instead of . And then, move the policy in the direction of the gradient of Q. This deterministic policy gradient is a special form of stochastic policy gradient.

  • Off-Policy Deterministic Actor-Critic (OPDAC)
    As in the case of stochastic policy gradient, off-policy is required to ensure adequate exploration. We can use Q-learning to train the critic. Deterministic policy removes the need for integral of actions and Q-learning removes the need for importance sampling.

  • Compatible Off-Policy Deterministic Actor-Critic (COPDAC)
    Since function approximator may not follow the true gradient, this paper suggest two restriction for the compatible action-value function.

    1. w minimize MSE of $$\epsilon(s;\theta,w)=\nabla_a Q^w(s,a) {a=\mu{\theta}(s)} - \nabla_a Q^{\mu}(s,a) {a=\mu{\theta}(s)}$$\

The resulting algorithm is called compatible off-policy deterministic actor-critic(COPDAC). We can use baseline function to reduce the variance of gradient estimator. If we use gradient Q-learning for critic, the algorithm is called COPDAC-GQ.


Great reviews of policy gradient algorithms.

Silver, David, et al. “Deterministic policy gradient algorithms.” ICML. 2014.