Explainer

What the words mean in MLflow, Metaflow, Argo Workflows, and Kubeflow

Ask four engineers, each of whom uses a different one of these tools, what a "run" is, and you will get four correct answers. The tools share a vocabulary the way neighbouring countries share a border: mostly by accident, and with strong views about where it falls. This post is a translation guide. It covers the words that cause the most confusion, says what each tool means by them, and ends with how Fendlemark sorts them out. What it says about the four tools comes from their own documentation, which is linked at the end.

1. One distinction explains most of it

MLflow records. The other three execute.

MLflow does not run your code. Your training script calls its tracking API, and MLflow stores what it is told: parameters, metrics, files. Metaflow, Argo Workflows, and Kubeflow Pipelines are the opposite. Each holds a definition of some work, starts it, and follows it to the end. So when someone says "the run failed", the first thing to settle is which kind of thing they mean: the execution, or the record of the execution. They are different objects with different ids, and either can be wrong while the other is fine. A training pod that is killed for exceeding its memory limit never gets to tell MLflow, so the MLflow run says RUNNING for as long as anyone looks, while the Argo Workflow that owned the pod says Failed. The post on running MLflow and Argo Workflows together shows how to find the stragglers.

2. The translation table

Each row is a job that any ML platform has to do. Each cell is what the tool calls the thing that does it. A dash means the tool has no such concept, which is a statement about its vocabulary and not about its quality.

What you meanMLflowMetaflowArgo WorkflowsKubeflow Pipelines
A reusable definition of the work—FlowWorkflowTemplatePipeline
One execution of itRunRunWorkflowRun
One unit of work inside an executionChild run, if you make oneTaskNodeStep (the SDK calls it a task)
The definition of one unit of work—SteptemplateComponent
A group of executionsExperiment— (runs belong to their flow; tags group them)— (labels, by convention)Experiment
An execution that repeats—A flow deployed with @scheduleCronWorkflowRecurring run
A small input valueParam (recorded after the fact)ParameterParameterParameter
What an execution leaves behindArtifactArtifactArtifactOutput artifact
A trained modelModel, registered model, model version, alias— (an artifact)—— (an artifact)

Three things stand out. Two rows look like agreement. The small input value is real agreement, though MLflow's "param" is a record of what was used rather than an input the tool supplies. "What an execution leaves behind" is not, as the section on artifacts below explains. And the first two rows show the real split. Metaflow and Kubeflow Pipelines name the definition (flow, pipeline) and call the execution a run. Argo Workflows names the execution a Workflow and uses WorkflowTemplate for the definition. That is why "pipeline" and "workflow" are not interchangeable, however often they are used as if they were.

3. The words that bite

Run

MLflow describes runs as "executions of some piece of data science code", recorded with their parameters, metrics, and artifacts. Metaflow defines "an execution of the flow, which we call a run", and counts it a success if the final end step completes. Kubeflow Pipelines says that "a run is a single execution of a pipeline". Argo Workflows has no separate word for it: the Workflow is the execution, so in Argo the thing you submit and the thing that ran are the same object.

One job therefore turns up under several names. Take a Metaflow flow deployed to Argo Workflows, which logs to MLflow. It has one Workflow, one Metaflow run, and one MLflow run:

Argo Workflows   ml-training/branchflow-r8qcn
Metaflow         BranchFlow/argo-branchflow-r8qcn
MLflow           6f1d2c9a8b7e4f30a1c5d9e8b2a47c13

Metaflow's documentation says that on Argo Workflows it uses the workflow execution name, prefixed with argo-, as the run id, so the first two share a stem. MLflow generates its own id, which is why the other two have to be written into MLflow tags before anyone can search for them from there.

Experiment

Three meanings, in three tools. MLflow's documentation says that "an experiment groups together runs and models for a specific task". Kubeflow Pipelines calls an experiment "a workspace where you can try different configurations of your pipelines", which also groups runs. Katib, the hyperparameter tuning subproject of Kubeflow, uses the same word for a different kind of thing: an Experiment is the specification of a tuning job, with a search space, an objective metric, and a search algorithm, and it works by running Trials, each of which tests one set of values. Metaflow and Argo Workflows have no experiment at all.

The practical difference is this: an MLflow experiment cannot fail, and a Katib experiment can. If the word appears in a status report, find out which one it is.

Pipeline, workflow, and flow

All three name a graph of steps, and the tools disagree about when each is a definition and when it is an execution.

Kubeflow Pipelines then adds a layer, because it executes on Argo Workflows by default. One Kubeflow Pipelines run is also an Argo Workflow: the same execution, named once by each layer. When someone says "the pipeline failed", ask whether they mean the definition, which cannot fail, or an execution of it, and which layer's name for it they are using.

Template

Argo Workflows uses the word twice, and its own documentation warns about the collision. A lower-case template is a unit of instruction inside a Workflow, which the documentation says can be "loosely thought of as 'functions'". There are two groups of them: definition templates that do the work (container, script, resource, suspend, plugin, container set, and HTTP) and invocator templates that call other templates (steps and DAG). A WorkflowTemplate is something else, a stored definition of a whole Workflow. The two meet in the reference syntax: templateRef borrows one template from a WorkflowTemplate, and workflowTemplateRef creates an entire Workflow from one.

Katib adds a third: an Experiment's trialTemplate is "the template that defines the Trial". Argo's lower-case template is a function, its WorkflowTemplate is a stored program, and Katib's trialTemplate is the recipe for one tuning attempt.

Job

Kubernetes has a Job, a batch/v1 resource that runs pods to completion; the adapter walkthrough builds an adapter for them. In the backend APIs of Kubeflow Pipelines, a "job" is a recurring run. Katib's documentation talks about hyperparameter tuning jobs, meaning what an Experiment sets going. Three different things, none of which is a run, and the Keepers have stopped using the word.

Artifact

Four tools, four things.

Anything that promises to show your artifacts is therefore showing four kinds of thing. Fendlemark's adapters are honest about the difference: the mlflow adapter lists artifact names and sizes, the metaflow adapter lists names only, and neither carries the bytes. The adapters reference says so.

Model

MLflow alone has four layers that people call "the model", per its Model Registry documentation. A model is what you create by logging one with a flavor's log_model(). A registered model is a named entry in the registry that holds versions, aliases, and tags. A model version is a numbered entry under it. An alias is a movable name that points at one version, such as champion, which you can reference as models:/MyModel@champion. Older material, and some older deployments, use stages such as Staging and Production instead of aliases, which the MLflow project has been moving away from.

Metaflow and Argo Workflows have no model concept, and in Kubeflow Pipelines a model is an artifact like any other. So "promote the model" can mean moving an alias, which changes a row, or copying a file, which changes where something is. Ask which.

Kubeflow

Kubeflow's own site describes "subprojects", an "ecosystem", and "distributions" as separate things. The subprojects include Pipelines, Katib, Notebooks, Trainer, and the Spark Operator, among others. KServe is listed under the ecosystem, not as a subproject. So "we run Kubeflow" tells you only that something from that list is installed. Fendlemark's kubeflow adapter reads Pipelines, and optionally Katib experiments and Notebook servers; the adapters reference says exactly which.

Tag, label, annotation, parameter

These are the words you use when you connect one tool to another, so the differences are worth knowing.

The rule of thumb is to put the identifier where the other tool can read it: a tag on the MLflow run, a label on the Workflow, a parameter on the Kubeflow Pipelines run. That is precisely the list of places the correlation matrix draws on, and it is why the adapters reference says correlation needs "a tag, label, or parameter".

4. Status words

Even "did it work?" has four vocabularies.

ToolWhat it says about an execution
MLflowRUNNING, SCHEDULED, FINISHED, FAILED, KILLED
MetaflowTwo booleans in the client instead of a status: Run.finished and Run.successful
Argo WorkflowsPending, Running, Succeeded, Failed, Error
Kubeflow PipelinesPENDING, RUNNING, SUCCEEDED, SKIPPED, FAILED, CANCELING, CANCELED, PAUSED

The trap is "finished". In MLflow, FINISHED is the status of a run that ended normally, and a run that raised an exception is FAILED. In Metaflow, Run.finished is true for any run that completed, successful or not; the question "did it work?" is answered by Run.successful instead. A dashboard that shows a green "finished" has told you two different things depending on which tool it was reading.

5. How the lattice sorts it out

A console that shows four tools has to pick words. Fendlemark picks six for what a thing is, which it calls a kind, and five for how it is going, which it calls a phase. Each facet keeps its own tool's vocabulary in its fields, so nothing is lost; the kind and phase are what let one table list them all.

KindConsole viewWhat lands there
runRunsMLflow runs and Metaflow runs, and any correlated entity that includes one
workflowWorkflowsArgo Workflows and Kubeflow Pipelines runs
pipelinePipelinesKubeflow pipelines, Argo WorkflowTemplates and CronWorkflows, Metaflow flows
modelModelsMLflow registered models and versions
notebookNotebooksKubeflow Notebook servers

The sixth kind, artifact, has no view of its own.

The mapping settles several of the collisions above by decision rather than by argument. A Metaflow flow, an Argo WorkflowTemplate, and a Kubeflow pipeline are three words for one kind of thing, a stored definition, and Fendlemark calls all three pipeline. A Kubeflow Pipelines run and the Argo Workflow beneath it are two names for one execution, and both are workflow. When facets of different kinds merge, the entity takes the first kind in the order run, workflow, pipeline, model, artifact, notebook, so an Argo Workflow whose step logged an MLflow run is a run. That is why the Runs view is where most people live.

The phases are Pending, Running, Succeeded, Failed, and Unknown, the last for anything an adapter cannot place. MLflow's FINISHED, Argo's Succeeded, and Kubeflow Pipelines' SUCCEEDED all become Succeeded, which is why the status badge is the same colour in every panel.

Three habits that need no software

Further reading