How-to

Running MLflow and Argo Workflows together

By the Fendlemark Keepers · 24 October 2026

Argo Workflows is a good place to run training, and MLflow is a good place to record it. Used together without any care, they give every training job two identities in two UIs, and nothing that connects them. This guide makes each side point at the other. None of it needs Fendlemark; the last section explains what Fendlemark does with it if you run it.

By the end, every MLflow run started by a workflow step will carry the workflow's name, uid, namespace, and pod as tags, every workflow will carry the MLflow run id as an output parameter, and a later step will log into the same MLflow run rather than starting another. The examples use a Workflow named resnet50-train-… in the ml-training namespace and an MLflow tracking server reachable inside the cluster at http://mlflow.mlflow.svc:5000. Substitute your own.

1. Point every step at the tracking server

The MLflow client reads its tracking server from the MLFLOW_TRACKING_URI environment variable, so a step needs no MLflow configuration in code. If your tracking server uses MLflow's basic authentication, the client also reads MLFLOW_TRACKING_USERNAME and MLFLOW_TRACKING_PASSWORD; a bearer token goes in MLFLOW_TRACKING_TOKEN instead.

Keep the address in a ConfigMap and the credentials in a Secret, with keys named exactly like the variables, so each step can import both with envFrom:

$ kubectl -n ml-training create configmap mlflow-client --from-literal=MLFLOW_TRACKING_URI=http://mlflow.mlflow.svc:5000
$ kubectl -n ml-training create secret generic mlflow-credentials --from-literal=MLFLOW_TRACKING_USERNAME=argo-runner --from-literal=MLFLOW_TRACKING_PASSWORD="$MLFLOW_PASSWORD"

Changing the tracking server later is then one ConfigMap edit, not an edit to every template.

2. Give each step the workflow's identity

Argo substitutes template variables in a template's fields before the pod starts, including environment variable values. Four of them identify where a step is running:

Variable	Value
`{{workflow.name}}`	The Workflow's name, for example `resnet50-train-x7k2p` when it was submitted with `generateName: resnet50-train-`. Unique within its namespace while the Workflow exists.
`{{workflow.uid}}`	The Workflow's Kubernetes UID. Unique across namespaces and across time, so it still identifies this run after a Workflow with the same name has been deleted and submitted again.
`{{workflow.namespace}}`	The namespace. With the name, enough to find the Workflow in the Argo UI or the CLI.
`{{pod.name}}`	The step's pod, available in container and script templates. Tells you which node of the workflow logged the run.

Here is a two-step Workflow that trains and then evaluates. The training step gets the tracking server from the ConfigMap and Secret, its identity from template variables, and declares an output parameter read from a file the script will write:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: resnet50-train-
  namespace: ml-training
spec:
  entrypoint: main
  arguments:
    parameters:
      - name: lr
        value: "0.0003"
  templates:
    - name: main
      steps:
        - - name: train
            template: train
        - - name: evaluate
            template: evaluate
            arguments:
              parameters:
                - name: mlflow-run-id
                  value: "{{steps.train.outputs.parameters.mlflow-run-id}}"

    - name: train
      outputs:
        parameters:
          - name: mlflow-run-id
            valueFrom:
              path: /tmp/mlflow-run-id
      container:
        image: registry.example.internal/vision/train:2026.10
        command: [python, train.py]
        args: ["--lr", "{{workflow.parameters.lr}}"]
        envFrom:
          - configMapRef:
              name: mlflow-client
          - secretRef:
              name: mlflow-credentials
        env:
          - name: ARGO_WORKFLOW_NAME
            value: "{{workflow.name}}"
          - name: ARGO_WORKFLOW_UID
            value: "{{workflow.uid}}"
          - name: ARGO_WORKFLOW_NAMESPACE
            value: "{{workflow.namespace}}"
          - name: ARGO_POD_NAME
            value: "{{pod.name}}"

    - name: evaluate
      inputs:
        parameters:
          - name: mlflow-run-id
      container:
        image: registry.example.internal/vision/train:2026.10
        command: [python, evaluate.py]
        envFrom:
          - configMapRef:
              name: mlflow-client
          - secretRef:
              name: mlflow-credentials
        env:
          - name: MLFLOW_RUN_ID
            value: "{{inputs.parameters.mlflow-run-id}}"

The quotes around each {{…}} value matter: unquoted, YAML would read the braces as a mapping.

3. Tag the MLflow run with that identity

MLflow tags are free-form string pairs on a run, searchable and shown in the run's page. mlflow.start_run accepts them directly, so the run carries the workflow's identity from the moment it exists:

import argparse
import os

import mlflow


def argo_tags():
    """Where this run came from, as MLflow tags."""
    return {
        "argo.workflow.name": os.environ["ARGO_WORKFLOW_NAME"],
        "argo.workflow.uid": os.environ["ARGO_WORKFLOW_UID"],
        "argo.workflow.namespace": os.environ["ARGO_WORKFLOW_NAMESPACE"],
        "argo.pod.name": os.environ["ARGO_POD_NAME"],
    }


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--lr", type=float, required=True)
    args = parser.parse_args()

    mlflow.set_experiment("resnet50")
    with mlflow.start_run(run_name=os.environ["ARGO_WORKFLOW_NAME"], tags=argo_tags()) as run:
        # Hand the run id back to Argo: this file is the step's output parameter.
        with open("/tmp/mlflow-run-id", "w") as f:
            f.write(run.info.run_id)

        mlflow.log_param("lr", args.lr)
        for epoch, metrics in enumerate(train(args.lr)):  # train() is your own loop
            mlflow.log_metrics(metrics, step=epoch)


if __name__ == "__main__":
    main()

A few choices in there are deliberate:

The run is named after the Workflow. MLflow does not require run names to be unique, and a name that matches the Argo UI makes the two easy to match by eye.
The run id is written first. Writing it before training means the file exists however long training takes, and the Argo UI shows it in the step's outputs once the step finishes.
The tags use the names argo.workflow.name and argo.workflow.uid. Any names would do for MLflow. These are the ones Fendlemark reads, as the last section explains.

Container images rarely include a .git directory, so MLflow usually cannot detect the commit it would otherwise record as mlflow.source.git.commit. If you bake the commit into the image at build time, set that tag yourself, along with mlflow.source.git.repoURL, in the same dictionary.

4. Log later steps into the same run

The evaluate step receives the training run's id as an input parameter and exposes it as MLFLOW_RUN_ID. When that variable is set, mlflow.start_run() resumes the run with that id instead of creating a new one, so evaluation metrics land next to the training metrics:

import mlflow

# MLFLOW_RUN_ID is set by the workflow, so this resumes the training run.
with mlflow.start_run() as run:
    mlflow.log_metrics(evaluate(run.info.run_id))  # evaluate() is your own code

One MLflow run per workflow is easier to compare across workflows than one per step, and it keeps the correlation simple: one workflow, one run. Steps that genuinely produce separate results, such as a hyperparameter sweep fanned out with withItems, are better as separate runs; pass the parent's id as parent_run_id to mlflow.start_run to nest them under it.

5. Find one from the other

From a workflow to its MLflow run. The train node's outputs in the Argo UI show mlflow-run-id. To go the other way round without the UI, look the run up by the workflow's uid, which is unambiguous even when names repeat:

$ kubectl -n ml-training get workflow resnet50-train-x7k2p -o jsonpath='{.metadata.uid}'
51d0c8e2-7b3a-4e19-a6f4-0c9d2e8b7a13

import mlflow

runs = mlflow.search_runs(
    filter_string="tags.`argo.workflow.uid` = '51d0c8e2-7b3a-4e19-a6f4-0c9d2e8b7a13'",
    search_all_experiments=True,
)
print(runs[["run_id", "status", "tags.argo.pod.name"]])

Tag names that contain dots must be wrapped in backticks (or double quotes) in an MLflow filter, and the value in single quotes. The same filter works in the search box of the MLflow UI: tags.`argo.workflow.name` = 'resnet50-train-x7k2p'.

From an MLflow run to its workflow. The run's argo.workflow.namespace and argo.workflow.name tags are everything the Argo CLI needs:

$ argo get resnet50-train-x7k2p -n ml-training

In the Argo UI, the same Workflow is at /workflows/ml-training/resnet50-train-x7k2p on your Argo Server.

Things that will happen eventually

Names repeat. A Workflow submitted with a fixed metadata.name instead of generateName gets the same name every time it is deleted and resubmitted, and the same name can exist in two namespaces at once. Search by argo.workflow.uid when it matters which one you mean.
Retries make more runs. If the training template has a retryStrategy, every attempt is a new pod, and with the script above a new MLflow run with a different argo.pod.name. Templates with a retryStrategy can also use {{retries}}, the attempt number, which is worth adding as a tag of its own.
Killed pods leave running runs. The with mlflow.start_run() block marks a run FAILED when Python raises, but a pod killed for exceeding its memory limit never reaches the end of the block, and its run stays RUNNING. Search for attributes.status = 'RUNNING' runs whose workflow has finished to find them.
The tracking server is a dependency. If MLflow is down, start_run fails once the client has exhausted its retries, and so does the step. That is usually what you want for training; if it is not, catch the error and decide explicitly.

How Fendlemark correlates them

If you run Fendlemark with the mlflow and argo adapters enabled, the tags above are all it needs. The argo adapter gives every Workflow facet the correlation key argo.workflow.name, and argo.workflow.uid alongside it. The mlflow adapter reads the run tags listed in correlation_tags, whose default list includes argo.workflow.name (and fendlemark.argo_workflow, which it treats the same way). When a Workflow facet and an MLflow run facet carry the same value, and their start times are no further apart than lattice.correlation.window, the lattice merges them into one entity:

$ fendlemark lattice inspect --key argo.workflow.name=resnet50-train-x7k2p
ID                   KIND  STATUS     FACETS       STARTED               TITLE
lt_459baa2c413ad692  run   Succeeded  argo+mlflow  2026-10-22T09:00:00Z  resnet50-train-x7k2p
$ fendlemark lattice explain lt_459baa2c413ad692
lt_459baa2c413ad692  run  Succeeded  "resnet50-train-x7k2p"
  facet  mlflow     6f1d2c9a8b7e4f30a1c5d9e8b2a47c13
  facet  argo       ml-training/resnet50-train-x7k2p
  merge  argo.workflow.name = "resnet50-train-x7k2p" linked argo/ml-training/resnet50-train-x7k2p and mlflow/6f1d2c9a8b7e4f30a1c5d9e8b2a47c13 (gap 14s)

Three details of the rules are worth knowing when you design your workflows:

The window is measured between start times: the Workflow's start and the MLflow run's start. The default, 90 seconds, suits a workflow whose tracked step comes first. If the step that opens the MLflow run follows a long data-preparation step, widen it to cover that step:
```
lattice:
  correlation:
    window: 20m
```
Two facets from the same adapter never merge. A workflow is merged with one MLflow run: the first to start after it. That is why step 4 resumes the training run instead of starting a second one. Extra runs carrying the same tag, such as retried attempts, stay entities of their own, and fendlemark lattice inspect --key argo.workflow.name=… lists all of them.
To correlate on the uid instead of the name, list it in correlation_tags. The list replaces the default rather than adding to it, so name every tag you want read:
```
adapters:
  mlflow:
    correlation_tags:
      - argo.workflow.uid
      - mlflow.source.git.commit
      - mlflow.source.git.repoURL
```
The argo adapter already provides argo.workflow.uid for every Workflow, so nothing changes on that side. Do this if the same Workflow name can start in two namespaces within the window.

The weak keys are a separate mechanism. If the MLflow run carries mlflow.source.git.commit and mlflow.source.git.repoURL, and the Workflow carries the same values in the fendlemark.io/git-sha and fendlemark.io/git-repo annotations, Fendlemark relates entities built from the same commit without merging them. The architecture page describes both kinds of key, and the correlation matrix shows which keys connect which adapters.