Explainer
What the words mean in MLflow, Metaflow, Argo Workflows, and Kubeflow
Ask four engineers, each of whom uses a different one of these tools, what a "run" is, and you will get four correct answers. The tools share a vocabulary the way neighbouring countries share a border: mostly by accident, and with strong views about where it falls. This post is a translation guide. It covers the words that cause the most confusion, says what each tool means by them, and ends with how Fendlemark sorts them out. What it says about the four tools comes from their own documentation, which is linked at the end.
1. One distinction explains most of it
MLflow records. The other three execute.
MLflow does not run your code. Your training script calls its tracking API, and MLflow stores what it is told: parameters, metrics, files. Metaflow, Argo Workflows, and Kubeflow Pipelines are the opposite. Each holds a definition of some work, starts it, and follows it to the end. So when someone says "the run failed", the first thing to settle is which kind of thing they mean: the execution, or the record of the execution. They are different objects with different ids, and either can be wrong while the other is fine. A training pod that is killed for exceeding its memory limit never gets to tell MLflow, so the MLflow run says RUNNING for as long as anyone looks, while the Argo Workflow that owned the pod says Failed. The post on running MLflow and Argo Workflows together shows how to find the stragglers.
2. The translation table
Each row is a job that any ML platform has to do. Each cell is what the tool calls the thing that does it. A dash means the tool has no such concept, which is a statement about its vocabulary and not about its quality.
| What you mean | MLflow | Metaflow | Argo Workflows | Kubeflow Pipelines |
|---|---|---|---|---|
| A reusable definition of the work | — | Flow | WorkflowTemplate | Pipeline |
| One execution of it | Run | Run | Workflow | Run |
| One unit of work inside an execution | Child run, if you make one | Task | Node | Step (the SDK calls it a task) |
| The definition of one unit of work | — | Step | template | Component |
| A group of executions | Experiment | — (runs belong to their flow; tags group them) | — (labels, by convention) | Experiment |
| An execution that repeats | — | A flow deployed with @schedule | CronWorkflow | Recurring run |
| A small input value | Param (recorded after the fact) | Parameter | Parameter | Parameter |
| What an execution leaves behind | Artifact | Artifact | Artifact | Output artifact |
| A trained model | Model, registered model, model version, alias | — (an artifact) | — | — (an artifact) |
Three things stand out. Two rows look like agreement. The small input value is real agreement, though MLflow's "param" is a record of what was used rather than an input the tool supplies. "What an execution leaves behind" is not, as the section on artifacts below explains. And the first two rows show the real split. Metaflow and Kubeflow Pipelines name the definition (flow, pipeline) and call the execution a run. Argo Workflows names the execution a Workflow and uses WorkflowTemplate for the definition. That is why "pipeline" and "workflow" are not interchangeable, however often they are used as if they were.
3. The words that bite
Run
MLflow describes runs as "executions of some piece of data science code", recorded with their parameters, metrics, and artifacts. Metaflow defines "an execution of the flow, which we call a run", and counts it a success if the final end step completes. Kubeflow Pipelines says that "a run is a single execution of a pipeline". Argo Workflows has no separate word for it: the Workflow is the execution, so in Argo the thing you submit and the thing that ran are the same object.
One job therefore turns up under several names. Take a Metaflow flow deployed to Argo Workflows, which logs to MLflow. It has one Workflow, one Metaflow run, and one MLflow run:
Argo Workflows ml-training/branchflow-r8qcn
Metaflow BranchFlow/argo-branchflow-r8qcn
MLflow 6f1d2c9a8b7e4f30a1c5d9e8b2a47c13Metaflow's documentation says that on Argo Workflows it uses the workflow execution name, prefixed with argo-, as the run id, so the first two share a stem. MLflow generates its own id, which is why the other two have to be written into MLflow tags before anyone can search for them from there.
Experiment
Three meanings, in three tools. MLflow's documentation says that "an experiment groups together runs and models for a specific task". Kubeflow Pipelines calls an experiment "a workspace where you can try different configurations of your pipelines", which also groups runs. Katib, the hyperparameter tuning subproject of Kubeflow, uses the same word for a different kind of thing: an Experiment is the specification of a tuning job, with a search space, an objective metric, and a search algorithm, and it works by running Trials, each of which tests one set of values. Metaflow and Argo Workflows have no experiment at all.
The practical difference is this: an MLflow experiment cannot fail, and a Katib experiment can. If the word appears in a status report, find out which one it is.
Pipeline, workflow, and flow
All three name a graph of steps, and the tools disagree about when each is a definition and when it is an execution.
- In Kubeflow Pipelines a pipeline is a definition: a Python function that is compiled to IR YAML and not run directly. What executes is a run.
- In Metaflow a flow is a definition, a graph of steps, and a run is an execution of it.
- In Argo Workflows a Workflow is both. Its documentation calls it a "live" object that holds the definition of the work and the state of its execution. The stored, reusable definition is a WorkflowTemplate, which the documentation describes as "definitions of Workflows that live in your cluster". A CronWorkflow is, in the documentation's words, "Workflow + some specific cron options".
Kubeflow Pipelines then adds a layer, because it executes on Argo Workflows by default. One Kubeflow Pipelines run is also an Argo Workflow: the same execution, named once by each layer. When someone says "the pipeline failed", ask whether they mean the definition, which cannot fail, or an execution of it, and which layer's name for it they are using.
Template
Argo Workflows uses the word twice, and its own documentation warns about the collision. A lower-case template is a unit of instruction inside a Workflow, which the documentation says can be "loosely thought of as 'functions'". There are two groups of them: definition templates that do the work (container, script, resource, suspend, plugin, container set, and HTTP) and invocator templates that call other templates (steps and DAG). A WorkflowTemplate is something else, a stored definition of a whole Workflow. The two meet in the reference syntax: templateRef borrows one template from a WorkflowTemplate, and workflowTemplateRef creates an entire Workflow from one.
Katib adds a third: an Experiment's trialTemplate is "the template that defines the Trial". Argo's lower-case template is a function, its WorkflowTemplate is a stored program, and Katib's trialTemplate is the recipe for one tuning attempt.
Job
Kubernetes has a Job, a batch/v1 resource that runs pods to completion; the adapter walkthrough builds an adapter for them. In the backend APIs of Kubeflow Pipelines, a "job" is a recurring run. Katib's documentation talks about hyperparameter tuning jobs, meaning what an Experiment sets going. Three different things, none of which is a run, and the Keepers have stopped using the word.
Artifact
Four tools, four things.
- In MLflow an artifact is an output file of a run, such as model weights or an image, kept in an artifact store. You choose the path and the format.
- In Metaflow, "data artifacts are created simply by assigning values to instance variables". Assign
self.modelin a step, and Metaflow persists it and hands it to later steps. You never name a file. - In Kubeflow Pipelines an output artifact is an output emitted by a component that the Kubeflow Pipelines UI can render.
- In Argo Workflows an artifact is a file or directory that one step produces and a later step consumes, kept in an artifact repository. Small values travel as parameters instead.
Anything that promises to show your artifacts is therefore showing four kinds of thing. Fendlemark's adapters are honest about the difference: the mlflow adapter lists artifact names and sizes, the metaflow adapter lists names only, and neither carries the bytes. The adapters reference says so.
Model
MLflow alone has four layers that people call "the model", per its Model Registry documentation. A model is what you create by logging one with a flavor's log_model(). A registered model is a named entry in the registry that holds versions, aliases, and tags. A model version is a numbered entry under it. An alias is a movable name that points at one version, such as champion, which you can reference as models:/MyModel@champion. Older material, and some older deployments, use stages such as Staging and Production instead of aliases, which the MLflow project has been moving away from.
Metaflow and Argo Workflows have no model concept, and in Kubeflow Pipelines a model is an artifact like any other. So "promote the model" can mean moving an alias, which changes a row, or copying a file, which changes where something is. Ask which.
Kubeflow
Kubeflow's own site describes "subprojects", an "ecosystem", and "distributions" as separate things. The subprojects include Pipelines, Katib, Notebooks, Trainer, and the Spark Operator, among others. KServe is listed under the ecosystem, not as a subproject. So "we run Kubeflow" tells you only that something from that list is installed. Fendlemark's kubeflow adapter reads Pipelines, and optionally Katib experiments and Notebook servers; the adapters reference says exactly which.
Tag, label, annotation, parameter
These are the words you use when you connect one tool to another, so the differences are worth knowing.
- An MLflow tag is a key-value pair on a run. Tags can be searched.
- Metaflow tags come in two kinds. User tags can be added, removed, and replaced. System tags are assigned by Metaflow and cannot be changed. The client's
tagsreturns both. - An Argo Workflows label is a Kubernetes label: it can be selected on, and its value is limited to 63 characters and a restricted character set. An annotation is for metadata you do not select on, and its values are not held to that limit.
- A Kubeflow Pipelines parameter is an input to a pipeline run, which makes a run's parameters a common place to pass another tool's run id into it.
The rule of thumb is to put the identifier where the other tool can read it: a tag on the MLflow run, a label on the Workflow, a parameter on the Kubeflow Pipelines run. That is precisely the list of places the correlation matrix draws on, and it is why the adapters reference says correlation needs "a tag, label, or parameter".
4. Status words
Even "did it work?" has four vocabularies.
| Tool | What it says about an execution |
|---|---|
| MLflow | RUNNING, SCHEDULED, FINISHED, FAILED, KILLED |
| Metaflow | Two booleans in the client instead of a status: Run.finished and Run.successful |
| Argo Workflows | Pending, Running, Succeeded, Failed, Error |
| Kubeflow Pipelines | PENDING, RUNNING, SUCCEEDED, SKIPPED, FAILED, CANCELING, CANCELED, PAUSED |
The trap is "finished". In MLflow, FINISHED is the status of a run that ended normally, and a run that raised an exception is FAILED. In Metaflow, Run.finished is true for any run that completed, successful or not; the question "did it work?" is answered by Run.successful instead. A dashboard that shows a green "finished" has told you two different things depending on which tool it was reading.
5. How the lattice sorts it out
A console that shows four tools has to pick words. Fendlemark picks six for what a thing is, which it calls a kind, and five for how it is going, which it calls a phase. Each facet keeps its own tool's vocabulary in its fields, so nothing is lost; the kind and phase are what let one table list them all.
| Kind | Console view | What lands there |
|---|---|---|
run | Runs | MLflow runs and Metaflow runs, and any correlated entity that includes one |
workflow | Workflows | Argo Workflows and Kubeflow Pipelines runs |
pipeline | Pipelines | Kubeflow pipelines, Argo WorkflowTemplates and CronWorkflows, Metaflow flows |
model | Models | MLflow registered models and versions |
notebook | Notebooks | Kubeflow Notebook servers |
The sixth kind, artifact, has no view of its own.
The mapping settles several of the collisions above by decision rather than by argument. A Metaflow flow, an Argo WorkflowTemplate, and a Kubeflow pipeline are three words for one kind of thing, a stored definition, and Fendlemark calls all three pipeline. A Kubeflow Pipelines run and the Argo Workflow beneath it are two names for one execution, and both are workflow. When facets of different kinds merge, the entity takes the first kind in the order run, workflow, pipeline, model, artifact, notebook, so an Argo Workflow whose step logged an MLflow run is a run. That is why the Runs view is where most people live.
The phases are Pending, Running, Succeeded, Failed, and Unknown, the last for anything an adapter cannot place. MLflow's FINISHED, Argo's Succeeded, and Kubeflow Pipelines' SUCCEEDED all become Succeeded, which is why the status badge is the same colour in every panel.
Three habits that need no software
- Name the tool. "The MLflow run", "the Argo Workflow", "the Kubeflow pipeline". Use the bare word only when the listener is looking at the same console as you.
- Say whether you mean the definition or an execution. It resolves most disagreements about "the pipeline" before they begin.
- Say which id you are quoting. A run id from one tool is not a run id in another, and the first thing the other tool will do with it is not find it.
Further reading
- A field guide to ML platform buzzwords, the companion to this post, on the words that describe products rather than tools
- Architecture: the lattice and The console: views, for the kinds and where each appears
- Adapters, for what each adapter reads and which keys it correlates on
- MLflow: Tracking and Model Registry
- Metaflow: Basics, Client API, and Scheduling with Argo Workflows
- Argo Workflows: Core Concepts, Workflow Templates, and Cron Workflows
- Kubeflow Pipelines: Concepts and Kubeflow architecture