Loss landscape geometry reveals stagewise development of transformers
Authors
Affiliations
George Wang
Timaeus Matthew Farrugia-Roberts Timaeus Jesse Hoogland Timaeus Liam Carroll Timaeus Susan Wei University of Melbourne Daniel Murfet University of Melbourne Published
Jun 16, 2024Links
Abstract
The development of the internal structure of neural networks throughout training occurs in tandem with changes in the local geometry of the population loss. By quantifying the degeneracy of this geometry using the recently proposed Local Learning Coefficient, we show that the training process for a transformer language model can be decomposed into discrete developmental stages. We connect these stages to interpretable shifts in input–output behavior and developments in internal structure. These findings offer new insights into transformer development and underscore the crucial role of loss landscape geometry in understanding the dynamics of deep learning.