University of Alberta Campus (CSC-333)
Presenter: Ehsan Imani, University of Alberta
Title: Representation Alignment in Neural Networks
Abstract: It is now a standard for neural network representations to be trained on large, publicly available datasets, and used for new problems. The reasons for why neural network representations have been so successful for transfer, however, are still not fully understood. We show that, after training, neural network representations align their top singular vectors to the targets. We investigate this representation alignment phenomenon in a variety of neural network architectures and find that (a) alignment emerges across a variety of different architectures and optimizers, with more alignment arising from depth (b) alignment increases for layers closer to the output and (c) existing high-performance deep CNNs exhibit high levels of alignment. We then highlight why alignment between the top singular vectors and the targets can speed up learning and show in a classic synthetic transfer problem that representation alignment correlates with positive and negative transfer to similar and dissimilar tasks.
Presenter Bio: Ehsan Imani is a PhD student at the University of Alberta. He received an MSc degree in computer science from the University of Alberta in 2019. Afterwards, he worked as a Research Assistant at the University of Alberta and later at Huawei Noah's Ark Lab. His research interests are Representation Learning and Reinforcement Learning.
The University of Alberta Artificial Intelligence (AI) Seminar is a weekly meeting where researchers (including students, developers, and professors) interested in AI can share their current research. Presenters include local speakers from the University of Alberta and industry as well as other institutions. The seminars discuss a wide range of topics related in any way to Artificial Intelligence, from foundational theoretical work to innovative applications of AI techniques to new fields and problems of interest. Learn more at the AI Seminar website and by subscribing to the mailing list!
Looking to build AI capacity? Need a speaker at your event?
