pytorch优化器Adam和Adamw中的weight_decay的区别
只介绍二者的weight_decay的区别
一般的梯度下降的方法是 θ j=θ j ? α ? ? θ j J heta_j= heta_j-\alpha\frac{\partial}{\partial heta_j}J θj?=θj??α?θj???J
算法
给出pytorch.optim.Adam类中具体实现的算法，来自pytorch中class Adam(Optimizer)
input : γ (lr) , β 1 , β 2 (betas) , θ 0 (params) , f ( θ ) (objective) λ (weight decay) , ? a m s g r a d initialize : m 0 ← 0 ( first moment) , v 0 ← 0 (second moment) , ? v 0 ^ m a x ← 0 for ? t=1 ? to ? … ? do g t ← ? θ f t ( θ t ? 1 ) if ? λ ≠ 0 g t ← g t + λ θ t ? 1 m t ← β 1 m t ? 1 + ( 1 ? β 1 ) g t v t ← β 2 v t ? 1 + ( 1 ? β 2 ) g t 2 m t ^ ← m t / ( 1 ? β 1 t ) v t ^ ← v t / ( 1 ? β 2 t ) if ? a m s g r a d v t ^ m a x ← m a x ( v t ^ m a x , v t ^ ) θ t ← θ t ? 1 ? γ m t ^ / ( v t ^ m a x + ? ) else θ t ← θ t ? 1 ? γ m t ^ / ( v t ^ + ? ) r e t u r n ? θ t \begin{aligned} &\rule{110mm}{0.4pt} \\ & extbf{input} : \gamma ext{ (lr)}, \beta_1, \beta_2 ext{ (betas)}, heta_0 ext{ (params)},f( heta) ext{ (objective)} \\ &\hspace{13mm} \lambda ext{ (weight decay)}, \: amsgrad \\ & extbf{initialize} : m_0 \leftarrow 0 ext{ ( first moment)}, v_0\leftarrow 0 ext{ (second moment)},\: \widehat{v_0}^{max}\leftarrow 0\\[-1.ex] &\rule{110mm}{0.4pt} \\ & extbf{for} \: t=1 \: extbf{to} \: \ldots \: extbf{do} \\ &\hspace{5mm}g_t \leftarrow
abla_{ heta} f_t ( heta_{t-1}) \\ &\hspace{5mm} extbf{if} \: \lambda
eq 0 \\ &\hspace{10mm} g_t \leftarrow g_t + \lambda heta_{t-1} \\ &\hspace{5mm}m_t \leftarrow \beta_1 m_{t-1} + (1 - \beta_1) g_t \\ &\hspace{5mm}v_t \leftarrow \beta_2 v_{t-1} + (1-\beta_2) g^2_t \\ &\hspace{5mm}\widehat{m_t} \leftarrow m_t/\big(1-\beta_1^t \big) \\ &\hspace{5mm}\widehat{v_t} \leftarrow v_t/\big(1-\beta_2^t \big) \\ &\hspace{5mm} extbf{if} \: amsgrad \\ &\hspace{10mm}\widehat{v_t}^{max} \leftarrow \mathrm{max}(\widehat{v_t}^{max}, \widehat{v_t}) \\ &\hspace{10mm} heta_t \leftarrow heta_{t-1} - \gamma \widehat{m_t}/ \big(\sqrt{\widehat{v_t}^{max}} + \epsilon \big) \\ &\hspace{5mm} extbf{else} \\ &\hspace{10mm} heta_t \leftarrow heta_{t-1} - \gamma \widehat{m_t}/ \big(\sqrt{\widehat{v_t}} + \epsilon \big) \\ &\rule{110mm}{0.4pt} \\[-1.ex] &\bf{return} \: heta_t \\[-1.ex] &\rule{110mm}{0.4pt} \\[-1.ex] \end{aligned} ?input:γ (lr),β1?,β2? (betas),θ0? (params),f(θ) (objective)λ (weight decay),amsgradinitialize:m0?←0 ( first moment),v0?←0 (second moment),v0?
?
