[One-page summary] Understanding plasticity in neural networks (arxiv 2023) by Lyle et al.

● Summary: stabilizing the loss landscape is crucial to preserve plasticity

 

● Approach highlight

• They show abrupt task change can drive instability in optimizers and drive plasticity loss

Adam optimizer

When loss change suddenly, $\hat{m}_{t}$ is updated more aggressively than $\hat{v}_{t}$ and that makes $\hat{u}_{t}$  instability. This simple solution is increasing 𝜖

• To understand the impact of optimization method on plasticity loss, they compare gradient descent and random walk (gaussian perturbation) for optimization method on plasticity loss

• Smoother loss landscape is both easier to optimize and has been empirically observed to exhibit better generalization

●  Main results

Effect of architectural and optimization interventions on plasticity loss

Visualization of relationship between network width and plasticity loss

● Discussion

• Why smoother loss landscape has better generalization performance