The world we live in today cannot do without data science and machine learning. It is therefore not a surprise that we have seen the proliferation of ML and data science tools over the last few years. From workflow orchestration to model tracking, pipelines to model management and deployment, the advantages that MLOps engineers and data scientists see in the tools cannot be overemphasized. Therefore, it is important for you to get a better visibility into the differences and similarities between the major tools in the data science world.
Metaflow and MLflow are the two of the most popular open-source tools under the machine learning platforms umbrella. Because these platforms are the open-source category leaders, they are often compared against each other despite being quite different. Thus, we decided to deep-dive into the topic to help people in the field get a clearer picture of how these two platforms compare to one another.
In this article, you will learn about the similarities and significant differences between Metaflow and MLflow.
Metaflow and its Components
Metaflow is a Python library that helps teams to build production machine learning. It was developed at Netflix initially to improve the productivity of data scientists who build and maintain different types of machine learning models. Metaflow is a much more focused tool and as such, the major concepts within it revolve around pipelines and orchestration:
Flow: A flow is simply the smallest unit of computation that can be scheduled for execution. It defines a workflow that pulls data from an external source as input, processes it, and produces output data. To implement a flow, users need to subclass FlowSpec and implement steps as methods, parameters or data triggers. The flow code and its external dependencies are encapsulated in the execution environment.
Graph: Metaflow deduces a directed acyclic graph (DAG) based on the transitions between step functions. These transitions are necessary to ensure that the graph is parsed statically from the source code of the flow.
Step: A step can be defined as a checkpoint that provides fault tolerance for the system. Metaflow typically takes a snapshot of the data produced by a step and uses it as input to the subsequent steps. Therefore, if a step fails, it can be resumed without having to rerun the preceding steps. Decorators can be used to modify the behavior of a step. The body of a step is known as the step code.
Runtime (Scheduler): The runtime or scheduler executes a flow; that is, it executes and orchestrates tasks defined by steps in topological order. You can use the metaflow.client, a Python API, to access the results of runs.
Datastore: This is an object store where both data artifacts and code snapshots can be persisted. It can be accessible in all environments where the Metaflow code is executed.
MLflow and its Components
MLflow is an open-source framework developed by Databricks for managing the whole machine learning cycle from start to finish, from training to deployment. Among the functions it offers are capabilities such as model tracking, management, packaging, and centralized lifecycle stage transitions. Some of the components of MLflow include the following:
Tracking: While executing your machine learning code, there's an API and UI for logging parameters, code versions, metrics, and output files so you can visualize them later.
Project: They provide a common style for packaging reusable data science code; nonetheless, each project is a code directory or a Git repository that uses a descriptor file to indicate dependencies and how to run the code.
Models: MLflow models are a standard for distributing machine learning models in a variety of flavors. There are a number of tools available to assist with the deployment of various models. Each model is then saved as a directory with arbitrary files and an ML model description file that identifies the flavors in which it can be used.
Registry: This offers you a centralized model store, UI and set of APIs, to collaboratively manage the full lifecycle of your MLflow Model. It provides model lineage, model versioning, stage transitions, and annotations.
Similarities between Metaflow and MLflow
Metaflow and MLflow share a few things in common. Perhaps the first one is the fact that both tools are open source. That is, they are easily accessible and free to use by anyone. Beyond this, there are other similarities as highlighted below:
Both platforms are scalable and make use of Python. However, it is important to notice that they do it in different ways, which we discuss below.
Both can be used for tracking experiments and versioning models. In fact, Metaflow versions and tracks all your data automatically.
They both have user interfaces. In MLFlow, you can use the tracking UI to visualize, search and compare runs. The UI also allows you to download metadata or run artifacts for analysis in other tools. As for Metaflow, the platform recently added a user interface as a separate add service for monitoring your metaflow executions.
Major Differences between Metaflow and MLflow
As much as Metaflow and MLflow share similarities, the two platforms have several significant differences that we present below:
Metaflow was originally developed at Netflix to help you design your workflow, run it at scale, and deploy it to production, while MLflow was originally built by Databrick to help you manage the end-to-end machine learning lifecycle including packaging ML code, experiment tracking, model deployment and management.
Unlike Metaflow, MLflow cannot be used for pipeline orchestration. Infact, the primary focus of Metaflow are pipelines while the primary focus of MLflow is experimentation.
As it was mentioned above, both platforms use Python. Metaflow is pretty Python opinionated: it is completely built as a Python library. In the meantime, MLflow is language agnostic. It has a Python/R/Java and REST API. The project steps are defined as shell commands. Thus, MLFlow is somewhat decoupled from the programming language.
Summary
If you have a large team interested in large in-production use-cases and you've already got a good set of tools for most things, Metaflow is a great option for you. The power of Metaflow is in the fact that its approach is opinionated. It may not fit every use case, but when it does, it is powerful and simple to work with.
However, if you would like to standardize the whole ML workflow and help data scientists to organize themselves better around experiments and machine learning models, MLflow is a viable option.
Valohai as an Alternative for Metaflow and MLflow
[CAUTION: Opinions ahead] We didn't just write this article to help choose between Metaflow and MLflow; we think we've built a better alternative.
With Metaflow, you'll likely be looking at building production pipelines with it and supplementing other areas with tools such as BentoML (model deployment). This may be a good approach as you can adopt these parts as you need them, but ultimately adopting more open-source tools comes with more overhead.
Meanwhile, what if your team needs a tool that combines the values and benefits of Metaflow and MLflow together? How do you do tracking and versioning, pipeline orchestration, and model deployment, all in one platform? Valohai addresses both issues.
Valohai is a managed platform with a 2-week free trial period.
If you are interested in learning more, check out:
This article continues our series on common tools teams are comparing for various machine learning tasks. You can check out some previous Metaflow and MLflow comparison articles like: