{"id":781,"date":"2024-05-24T18:22:01","date_gmt":"2024-05-24T18:22:01","guid":{"rendered":"https:\/\/people.irisa.fr\/Remi.Gribonval\/?p=781"},"modified":"2024-05-24T18:22:04","modified_gmt":"2024-05-24T18:22:04","slug":"conservation-laws-for-neural-network-training-two-papers-at-neurips23-icml24","status":"publish","type":"post","link":"https:\/\/perso.ens-lyon.fr\/remi.gribonval\/conservation-laws-for-neural-network-training-two-papers-at-neurips23-icml24\/","title":{"rendered":"Conservation laws for neural network training: two papers at @NeurIPS23 &#038; @ICML24"},"content":{"rendered":"\n<p>We study conservation laws during the (euclidean or not) gradient or momentum flow of neural networks.<\/p>\n\n\n\n<p><a href=\"https:\/\/arxiv.org\/abs\/2405.12888\">Keep the Momentum: Conservation Laws beyond Euclidean Gradient Flows<\/a>, accepted at ICML24<\/p>\n\n\n\n<p>1\/ We define the <strong>concept of conservation laws for momentum flows<\/strong> and show how to extend the framework from our previous paper (<a href=\"https:\/\/arxiv.org\/abs\/2307.00144\">Abide by the Law and Follow the Flaw: Conservation Laws for Gradient Flows<\/a>, oral @NeurIPS23)<strong> <\/strong>for non-Euclidean gradient flow (GF) and momentum flow (MF) settings. <strong>In stark contrast to the case of GF, conservation laws for MF exhibit temporal dependence<\/strong>.<\/p>\n\n\n\n<p>2\/ We discover <strong>new conservation laws<\/strong> for linear networks in the Euclidean momentum case, and these new laws are complete. In contrast, <strong>there is no conservation law for ReLU networks in the Euclidean momentum case<\/strong>.\u00a0<\/p>\n\n\n\n<p>3\/ <strong>In a non-Euclidean context<\/strong>, such as in NMF or for ICNN implemented with two-layer ReLU networks, <strong>we discover new conservation laws for gradient flows<\/strong> and find none in the momentum case. We obtain n<strong>ew conservation laws in the Natural Gradient Flow case<\/strong>.<\/p>\n\n\n\n<p>4\/ We shed light on a quasi-systematic loss of conservation when transitioning from the GF to the MF setting.<\/p>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>We study conservation laws during the (euclidean or not) gradient or momentum flow of neural networks. Keep the Momentum: Conservation Laws beyond Euclidean Gradient Flows, accepted at ICML24 1\/ We define the concept of conservation laws for momentum flows and show how to extend the framework from our previous paper (Abide by the Law and &hellip; <\/p>\n<p><a class=\"more-link btn\" href=\"https:\/\/perso.ens-lyon.fr\/remi.gribonval\/conservation-laws-for-neural-network-training-two-papers-at-neurips23-icml24\/\">Continue reading<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[13],"tags":[],"class_list":["post-781","post","type-post","status-publish","format-standard","hentry","category-preprint","nodate","item-wrap"],"_links":{"self":[{"href":"https:\/\/perso.ens-lyon.fr\/remi.gribonval\/wp-json\/wp\/v2\/posts\/781","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/perso.ens-lyon.fr\/remi.gribonval\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/perso.ens-lyon.fr\/remi.gribonval\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/perso.ens-lyon.fr\/remi.gribonval\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/perso.ens-lyon.fr\/remi.gribonval\/wp-json\/wp\/v2\/comments?post=781"}],"version-history":[{"count":1,"href":"https:\/\/perso.ens-lyon.fr\/remi.gribonval\/wp-json\/wp\/v2\/posts\/781\/revisions"}],"predecessor-version":[{"id":782,"href":"https:\/\/perso.ens-lyon.fr\/remi.gribonval\/wp-json\/wp\/v2\/posts\/781\/revisions\/782"}],"wp:attachment":[{"href":"https:\/\/perso.ens-lyon.fr\/remi.gribonval\/wp-json\/wp\/v2\/media?parent=781"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/perso.ens-lyon.fr\/remi.gribonval\/wp-json\/wp\/v2\/categories?post=781"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/perso.ens-lyon.fr\/remi.gribonval\/wp-json\/wp\/v2\/tags?post=781"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}