{"id":774,"date":"2024-05-24T17:57:23","date_gmt":"2024-05-24T17:57:23","guid":{"rendered":"https:\/\/people.irisa.fr\/Remi.Gribonval\/?p=774"},"modified":"2024-05-24T17:57:24","modified_gmt":"2024-05-24T17:57:24","slug":"invited-talk-math-machine-learning-seminar-mpi-mis-ucla-online-may-16-2024","status":"publish","type":"post","link":"https:\/\/perso.ens-lyon.fr\/remi.gribonval\/invited-talk-math-machine-learning-seminar-mpi-mis-ucla-online-may-16-2024\/","title":{"rendered":"Invited talk, Math Machine Learning seminar MPI MIS + UCLA, online, May 16, 2024"},"content":{"rendered":"\n<p><a href=\"https:\/\/www.mis.mpg.de\/events\/event\/conservation-laws-for-gradient-flows\">\u2018Conservation Laws for Gradient Flows\u2019<\/a><\/p>\n\n\n\n<p>Understanding the geometric properties of gradient descent dynamics is<br>a key ingredient in deciphering the recent success of very large<br>machine learning models. A striking observation is that trained<br>over-parameterized models retain some properties of the optimization<br>initialization. This \u201cimplicit bias\u201d is believed to be responsible for<br>some favorable properties of the trained models and could explain<br>their good generalization properties. In this work, we expose the<br>definitions and properties of &#8220;conservation laws&#8221;, that define<br>quantities conserved during gradient flows of a given machine learning<br>model, such as a ReLU network, with any training data and any loss.<br>After explaining how to find the maximal number of independent<br>conservation laws via Lie algebra computations, we provide algorithms<br>to compute a family of polynomial laws, as well as to compute the<br>number of (not necessarily polynomial) conservation laws. We obtain<br>that on a number of architecture there are no more laws than the known<br>ones, and we identify new laws for certain flows with momentum and\/or<br>non-Euclidean geometries.<br>Joint work with Sibylle Marcotte and Gabriel Peyr\u00e9.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>\u2018Conservation Laws for Gradient Flows\u2019 Understanding the geometric properties of gradient descent dynamics isa key ingredient in deciphering the recent success of very largemachine learning models. A striking observation is that trainedover-parameterized models retain some properties of the optimizationinitialization. This \u201cimplicit bias\u201d is believed to be responsible forsome favorable properties of the trained models and &hellip; <\/p>\n<p><a class=\"more-link btn\" href=\"https:\/\/perso.ens-lyon.fr\/remi.gribonval\/invited-talk-math-machine-learning-seminar-mpi-mis-ucla-online-may-16-2024\/\">Continue reading<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4],"tags":[],"class_list":["post-774","post","type-post","status-publish","format-standard","hentry","category-talk","nodate","item-wrap"],"_links":{"self":[{"href":"https:\/\/perso.ens-lyon.fr\/remi.gribonval\/wp-json\/wp\/v2\/posts\/774","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/perso.ens-lyon.fr\/remi.gribonval\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/perso.ens-lyon.fr\/remi.gribonval\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/perso.ens-lyon.fr\/remi.gribonval\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/perso.ens-lyon.fr\/remi.gribonval\/wp-json\/wp\/v2\/comments?post=774"}],"version-history":[{"count":1,"href":"https:\/\/perso.ens-lyon.fr\/remi.gribonval\/wp-json\/wp\/v2\/posts\/774\/revisions"}],"predecessor-version":[{"id":775,"href":"https:\/\/perso.ens-lyon.fr\/remi.gribonval\/wp-json\/wp\/v2\/posts\/774\/revisions\/775"}],"wp:attachment":[{"href":"https:\/\/perso.ens-lyon.fr\/remi.gribonval\/wp-json\/wp\/v2\/media?parent=774"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/perso.ens-lyon.fr\/remi.gribonval\/wp-json\/wp\/v2\/categories?post=774"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/perso.ens-lyon.fr\/remi.gribonval\/wp-json\/wp\/v2\/tags?post=774"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}