Comparing Dynamics of Glasses with Deep Neural Networks
Time permitting, a second section on Deep Learning is presented. We study dynam- ics and energy (or loss) landscape of feedforward Deep Neural Networks , with an emphasis on their Mean Square Displacement, and find that they exhibit aging on a finite time scale. Later, they start diffusing at the bottom of the landscape. Further, we argue that the slow dynamics is due rather to flat directions in the landscape than to barrier crossing, and we show evidence of an under- to over-parametrized phase transition.
1.  M. Baity-Jesi, G. Biroli & C. Cammarota, J. Stat. Mech. (2018) 013301.
2.  C. Cammarota & E. Marinari, Phys. Rev. E 92 010301(R) (2015).
3.  M. Baity-Jesi, A. Achard-de Lustrac & G. Biroli, Phys. Rev. E 98, 012133 (2018).
4.  M. Baity-Jesi, L. Sagun, M. Geiger, S. Spigler, G. Ben-Arous, C. Cammarota, Y. LeCun, M. Wyart & G. Biroli, PMLR 80:324-333, 2018 (ICML 2018).