A flexible framework for composing end to end machine learning pipelines

Abstract

In this thesis, we aim to simplify the building of end-to-end machine learning pipelines while preserving the performance of such pipelines on real data. As a solution to this, we propose the MLBlocks framework, a system that allows an end user to obtain a pipeline with only data and a list of data science blocks. Once a pipeline is specified, a user can tune its hyper-parameters, as well as fit and predictions, with minimal code. When building MLBlocks, we first develop a data science block library that seamlessly integrates third party blocks without integration code, providing a foundation for users to start building data science pipelines. We then provide the MLPipeline framework that allows users to simply tie together these blocks and perform the aforementioned tuning, fitting, and predicting operations with the resulting pipelines. Finally, we test the framework's usability as well as its ability to preserve performance on real data by running several pipelines on various data modalities and by integrating MLBlocks into a larger scale project. Since we are able to replicate the pipelines already in use, we are able to obtain identical results while dramatically simplifying application logic. We conclude that MLBlocks succeeds in providing a simple but effective solution to making the construction of high-performing end-to-end pipelines both simpler and more accessible.