Quantifying Degeneracy in Singular Models via the Learning Coefficient
Authors
Affiliations
Edmund Lau University of Melbourne George Wang
Timaeus Daniel Murfet University of Melbourne Susan Wei University of Melbourne Published
Aug 23, 2023Links
Abstract
Deep neural networks (DNN) are singular statistical models which exhibit complex degeneracies. In this work, we illustrate how a quantity known as the learning coefficient introduced in singular learning theory quantifies precisely the degree of degeneracy in deep neural networks. Importantly, we will demonstrate that degeneracy in DNN cannot be accounted for by simply counting the number of 'flat' directions.