强化学习（三）—— 策略学习（Policy-Based）及策略梯度（Policy Gradient）
1. 策略学习
Policy Network
- 通过策略网络近似策略函数
π
(
a
∣
s
t
)
≈
π
(
a
∣
s
t
;
θ
)
π(a|s_t)≈π(a|s_t;theta)
π(a∣st)≈π(a∣st;θ) - 状态价值函数及其近似
V
π
(
s
t
)
=
∑
a
π
(
a
∣
s
t
)
Q
π
(
s
t
,
a
)
V_π(s_t)=sum_aπ(a|s_t)Q_π(s_t,a)
Vπ(st)=a∑π(a∣st)Qπ(st,a)
V
(
s
t
;
θ
)
=
∑
a
π
(
a
∣
s
t
;
θ
)
⋅
Q
π
(
s
t
,
a
)
V(s_t;theta)=sum_aπ(a|s_t;theta)·Q_π(s_t,a)
V(st;θ)=a∑π(a∣st;θ)⋅Qπ(st,a) - 策略学习最大化的目标函数
J
(
θ
)
=
E
S
[
V
(
S
;
θ
)
]
J(theta)=E_S[V(S;theta)]
J(θ)=ES[V(S;θ)] - 依据策略梯度上升进行
θ
←
θ
+
β
⋅
∂
V
(
s
;
θ
)
∂
θ
thetagetstheta+beta·frac{partial V(s;theta)}{partial theta}
θ←θ+β⋅∂θ∂V(s;θ)
2. 策略梯度
Policy Gradient
∂
V
(
s
;
θ
)
θ
=
∑
a
Q
π
(
s
,
a
)
∂
π
(
a
∣
s
;
θ
)
∂
θ
=
∫
a
Q
π
(
s
,
a
)
∂
π
(
a
∣
s
;
θ
)
∂
θ
=
∑
a
π
(
a
∣
s
;
θ
)
⋅
Q
π
(
s
,
a
)
∂
l
n
[
π
(
a
∣
s
;
θ
)
]
∂
θ
=
E
A
∼
π
(
a
∣
s
;
θ
)
[
Q
π
(
s
,
A
)
∂
l
n
[
π
(
A
∣
s
;
θ
)
]
∂
θ
]
≈
Q
π
(
s
t
,
a
t
)
∂
l
n
[
π
(
a
t
∣
s
t
;
θ
)
]
∂
θ
frac{partial V(s;theta)}{theta}=sum_a{Q_pi(s,a)frac{partialpi(a|s;theta)}{partialtheta}}\=int_a{Q_pi(s,a)frac{partialpi(a|s;theta)}{partialtheta}}\=sum_a{pi(a|s;theta)·Q_pi(s,a)frac{partial ln[pi(a|s;theta)]}{partialtheta}}\=E_{Asimpi(a|s;theta)}[Q_pi(s,A)frac{partial ln[pi(A|s;theta)]}{partialtheta}]\≈Q_pi(s_t,a_t)frac{partial ln[pi(a_t|s_t;theta)]}{partialtheta}
θ∂V(s;θ)=a∑Qπ(s,a)∂θ∂π(a∣s;θ)=∫aQπ(s,a)∂θ∂π(a∣s;θ)=a∑π(a∣s;θ)⋅Qπ(s,a)∂θ∂ln[π(a∣s;θ)]=EA∼π(a∣s;θ)[Qπ(s,A)∂θ∂ln[π(A∣s;θ)]]≈Qπ(st,at)∂θ∂ln[π(at∣st;θ)]
- 观测得到状态
s
t
s_t
st - 依据策略函数随机采样动作
a
t
=
π
(
a
t
∣
s
t
;
θ
)
a_t = pi(a_t|s_t;theta)
at=π(at∣st;θ) - 计算价值函数
q
t
=
Q
π
(
s
t
,
a
t
)
q_t = Q_pi(s_t,a_t)
qt=Qπ(st,at) - 求取策略网络的梯度
d
θ
,
t
=
∂
l
n
[
π
(
a
t
∣
s
t
;
θ
)
]
∂
θ
∣
θ
=
θ
t
d_{theta,t}=frac{partial ln[pi(a_t|s_t;theta)]}{partialtheta}|theta=theta_t
dθ,t=∂θ∂ln[π(at∣st;θ)]∣θ=θt - 计算近似的策略梯度
g
(
a
t
,
θ
t
)
=
q
t
⋅
d
θ
,
t
g(a_t,theta _t)=q_t·d_{theta,t}
g(at,θt)=qt⋅dθ,t - 更新策略网络
θ
t
+
1
=
θ
t
+
β
⋅
g
(
a
t
,
θ
t
)
theta_{t+1}=theta_t+beta·g(a_t,theta_t)
θt+1=θt+β⋅g(at,θt)
3. 案例
目前没有好的方法近似动作价值函数，则未撰写案例。
by CyrusMay 2022 03 29