Building Machine Learning Models like Open-Source Software with git-theta [Colin Raffel & Nikhil Kandpal]

Pre-trained models have become a cornerstone of machine learning thanks to the fact that they are often applicable to a huge range of downstream applications. However, these models are typically created by resource-rich research groups that unilaterally decide how a given model should be built, trained, and released, after which point it is never updated. In contrast, open-source development has demonstrated that it is possible for a community of contributors to work together to iteratively build complex and widely used software. This kind of large-scale distributed collaboration is made possible through a mature set of tools including version control and package management. This talk will discuss our research that aims to make it possible to build machine learning models in the way that open-source software is developed. After briefly discussing our work on merging models, model patches, and modular architectures, we will provide a thorough overview of git-theta, our version control system for model parameters. git-theta integrates into the standard git workflow and supports cheaply-communicable patches and can natively handle automatic merging. The talk will conclude with a brief demo of git-theta's functionality.