# Kubeflow and the Fendlemark Kubeflow adapter

> How Kubeflow and Kubeflow Pipelines organise, run, and record ML work, and how Fendlemark's read-only kubeflow adapter reads pipeline runs into the lattice.

This reference has two halves. The first explains Kubeflow: how the platform separates users, how Kubeflow Pipelines (KFP) turns Python into Argo Workflows and records what happened, and the REST API through which it is read. The second explains the `kubeflow` adapter in Fendlemark 0.9.12: what it requests and with which credentials, the facets and correlation keys it produces, and how the lattice joins a pipeline run to the Argo Workflow that executed it and the MLflow run it logged. Configuration, permissions, operations, a worked example, and a glossary follow.

The Kubeflow material covers platform releases from 1.9 (July 2024) to 26.03 (March 2026) and KFP 2.x up to 2.17.3 (October 2026). After 1.11 the platform moved to year-and-month version numbers. The adapter's summary is in the [adapters reference](https://fendlemark.app/docs/adapters/#kubeflow).

## How Kubeflow is organised

### Components and releases

Kubeflow is a set of separately developed Kubernetes projects. The [kubeflow/manifests](https://github.com/kubeflow/manifests) repository, now named `kubeflow/community-distribution`, tests and releases them together as one platform. Each component can also be installed alone, and KFP often is. The versions each release ships:

| Kubeflow | KFP | Katib | Training Operator | Trainer | KServe | Model Registry |
| --- | --- | --- | --- | --- | --- | --- |
| 1.9 | 2.2.0 | 0.17.0 | 1.8.0 | — | 0.13.0 | 0.2.1-alpha |
| 1.10 | 2.4.1 | 0.18.0 | 1.9.1 | — | 0.14.1 | 0.2.15.3 |
| 1.11 | 2.15.0 | 0.19.0 | 1.9.2 | 2.1.0 | 0.15.2 | 0.3.4 |
| 26.03.1 | 2.16.1 | 0.19.0 | 1.9.2 | 2.2.0 | 0.18.0 | 0.3.9, as Kubeflow Hub |

The platform also includes the Central Dashboard, Profiles, Notebooks, Istio, Dex, oauth2-proxy, cert-manager, and Knative. KFP releases more often than the platform, so a standalone installation can run 2.17.x while the newest platform release bundles 2.16.1.

### Profiles and namespaces

A `Profile` (`kubeflow.org/v1`, cluster-scoped) names an owner. The profile controller creates a namespace of the same name containing:

- a RoleBinding, `namespaceAdmin`, that grants the owner the `kubeflow-admin` ClusterRole;
- the ServiceAccounts `default-editor`, with `kubeflow-edit` permissions, and `default-viewer`, with Kubernetes `view`;
- an Istio AuthorizationPolicy, `ns-owner-access-istio`, that admits the owner's requests to workloads in the namespace.

Contributors added through the dashboard get `kubeflow-edit` or `kubeflow-view` RoleBindings. Those ClusterRoles are aggregates to which each component contributes rules; KFP's contributions are `kubeflow-pipelines-edit` and `kubeflow-pipelines-view`. Kubeflow's documentation states that a profile isolates no more than a Kubernetes namespace does.

KFP fits this model as follows:

- An experiment belongs to a namespace, and runs and recurring runs belong to the namespace of their experiment.
- A run executes as an Argo Workflow in that namespace, as `default-editor` unless `DEFAULTPIPELINERUNNERSERVICEACCOUNT` names another account.
- A pipeline definition is either namespaced or shared. A shared pipeline has an empty namespace and is visible from every profile.

### The Central Dashboard and authentication

The Central Dashboard is the web shell at the root of the Kubeflow host: a namespace selector and navigation around each component's own UI. The KFP UI lives under `/pipeline/`, with hash routes such as `/pipeline/#/runs/details/<run_id>`.

Since 1.9, requests pass through this chain:

1. **Login.** The Istio ingress gateway sends each request through oauth2-proxy, which signs the user in with Dex.
2. **Identity headers.** An Istio `RequestAuthentication`, `dex-jwt`, validates the Dex JWT. It copies the `email` claim into the `kubeflow-userid` header and the `groups` claim into `kubeflow-groups`.
3. **Programmatic clients.** The optional `istio-m2m` component adds a second `RequestAuthentication`, `m2m-token-issuer`. It accepts Kubernetes service account tokens at the gateway and maps their `sub` claim (`system:serviceaccount:<namespace>:<name>`) into `kubeflow-userid`.
4. **Authorisation.** A mesh-wide `global-deny-all` AuthorizationPolicy denies whatever no per-workload policy allows.

The policy on the KFP API server, `ml-pipeline`, allows two kinds of request: those from KFP's own service accounts, such as `ml-pipeline-ui` and `ml-pipeline-persistenceagent`, and any request without a `kubeflow-userid` header. A client inside the cluster that calls `ml-pipeline` directly must therefore omit the header and identify itself with a bearer token. The UI may forward the header because it runs as one of KFP's own principals.

## Kubeflow Pipelines

### Services

| Service | Role |
| --- | --- |
| `ml-pipeline` | The API server: REST on port 8888, gRPC on 8887. Stores pipelines, experiments, runs, and recurring runs in a relational database (MySQL in the default manifests). |
| `ml-pipeline-ui` | The web UI. It also proxies `/apis/...` to the API server. |
| `ml-pipeline-persistenceagent` | Watches Argo Workflows and reports their status to the API server. This is how a run's state changes. |
| `ml-pipeline-scheduledworkflow` | Controller for `ScheduledWorkflow` resources, which implement recurring runs. |
| `metadata-grpc-service` | ML Metadata (MLMD), the lineage store. |
| Argo workflow controller | Executes the Workflows. KFP installs only the controller (from Argo Workflows v4.0.6 at KFP 2.17.3), not Argo Server. |

Pipeline definitions are stored in the database by default, or as Kubernetes resources in recent releases. The health endpoint reports which, as `pipeline_store`.

### Authoring with the v2 SDK

A **component** is one containerised step. `@dsl.component` turns a Python function into a component, with `base_image` and `packages_to_install` describing its image, and `@dsl.container_component` wraps any image. Inputs and outputs are either **parameters** (`str`, `int`, `float`, `bool`, `list`, `dict`) or **artifacts**: files with metadata, declared as `Input[Dataset]`, `Output[Model]`, and so on. A **pipeline** (`@dsl.pipeline`) calls components to create **tasks** and wires outputs to inputs. It can branch and loop with `dsl.If`, `dsl.Elif`, `dsl.Else`, `dsl.ParallelFor`, and `dsl.ExitHandler`.

```python
from kfp import compiler, dsl

@dsl.component(base_image="python:3.12")
def add(a: float, b: float) -> float:
    return a + b

@dsl.pipeline(name="add-twice")
def add_twice(x: float = 1.0):
    first = add(a=x, b=x)
    add(a=first.output, b=x)

compiler.Compiler().compile(add_twice, package_path="add-twice.yaml")
```

The compiler writes [IR YAML](https://www.kubeflow.org/docs/components/pipelines/concepts/ir-yaml/), a serialised `PipelineSpec` made of `pipelineInfo`, the `root` DAG, `components`, `deploymentSpec`, `schemaVersion`, and `sdkVersion`. A pipeline that uses the `kfp-kubernetes` extension for secrets, volumes, or node selectors gets a second YAML document, a `PlatformSpec`.

Uploading IR YAML creates a **pipeline** and its first **pipeline version**, each with its own ID, and later uploads add versions. A run either references a version through `pipeline_version_reference` (`pipeline_id` and `pipeline_version_id`) or embeds the spec in `pipeline_spec`. The SDK's `create_run_from_pipeline_func` and `create_run_from_pipeline_package` embed it, so runs made that way have no pipeline ID.

Placeholders resolve to run metadata when a task executes:

| Placeholder | Resolves to |
| --- | --- |
| `dsl.PIPELINE_JOB_ID_PLACEHOLDER` | The run ID. |
| `dsl.PIPELINE_JOB_NAME_PLACEHOLDER` | The run's display name. |
| `dsl.PIPELINE_JOB_RESOURCE_NAME_PLACEHOLDER` | The name of the Argo Workflow. |
| `dsl.PIPELINE_TASK_ID_PLACEHOLDER` | The task ID. |

### Experiments, runs, and recurring runs

An **experiment** groups runs within a namespace; a run created without one goes to `Default`. A **run** is one execution of a pipeline with values for its parameters (`runtime_config.parameters`) and, optionally, a `pipeline_root`, the object-store prefix for its artifacts. It proceeds in four steps:

1. **Record.** The API server stores the run in state `PENDING`.
2. **Compile.** It compiles the IR into an Argo Workflow in the run's namespace. The Workflow uses `generateName: <pipeline name>-` and carries the label `pipeline/runid: <run ID>`.
3. **Execute.** Argo runs each task as two steps. A *driver* resolves inputs, checks the cache, and records the execution in MLMD. An *executor* runs the user's container under KFP's *launcher*, which uploads and records the outputs. Component pods carry the label `pipelines.kubeflow.org/v2_component: "true"`.
4. **Report.** The persistence agent reports Workflow status to the API server. Once the final state is stored, it labels the Workflow `pipeline/persistedFinalState`. The manifests set `TTL_SECONDS_AFTER_WORKFLOW_FINISH` to `86400`, so finished Workflows are removed after about a day. The KFP run record remains.

A **recurring run** starts runs on a trigger: a cron schedule (`cron_schedule.cron`) or a fixed period (`periodic_schedule.interval_second`). Cron takes six fields with seconds first, so `0 0 2 * * *` means 02:00 every day. `max_concurrency` (1 to 10) limits how many of its runs are active at once, `no_catchup` skips missed intervals, and `status` is `ENABLED` or `DISABLED`. A `ScheduledWorkflow` implements each recurring run, and every run it starts records `recurring_run_id`.

### Pipelines v1 and v2

The v2 backend has shipped with Kubeflow since 1.8. The differences matter when reading older installations:

| | KFP v1 | KFP v2 |
| --- | --- | --- |
| SDK output | Argo `Workflow` YAML, compiled by the SDK | IR YAML, compiled to a Workflow by the API server |
| Metadata | A metadata writer infers MLMD records from pods | Driver and launcher write MLMD directly |
| Caching | A mutating webhook keyed on the pod spec | A fingerprint of the task's definition and inputs, checked by the driver |
| API | `/apis/v1beta1`, with resource references for namespaces and experiments | `/apis/v2beta1`, with explicit `namespace` and `experiment_id` |

KFP 2.x still runs pipelines compiled and submitted with the v1 SDK. Recent releases can refuse them (`BLOCK_V1_PIPELINES`).

### Metadata, lineage, and caching

[MLMD](https://www.kubeflow.org/docs/components/pipelines/concepts/metadata/) records what each run did as a graph:

- **Artifacts** are typed records with a URI and properties, such as `system.Dataset`, `system.Model`, and `system.Metrics`.
- **Executions** record one task each, with a state such as `RUNNING`, `COMPLETE`, `FAILED`, or `CACHED`.
- **Events** are the edges between executions and the artifacts they read or wrote.
- **Contexts** group executions and artifacts: a pipeline context for the pipeline and a pipeline run context for each run. A run reports both IDs in `run_details`.

Following events back from an artifact gives its lineage, which the KFP UI draws.

[Caching](https://www.kubeflow.org/docs/components/pipelines/user-guides/core-functions/caching/) is on by default. Before a task runs, the driver fingerprints its definition and inputs. If an earlier execution has the same fingerprint and its outputs still exist, KFP reuses them and does not start the executor. The API's `SKIPPED` state covers such tasks. To control caching:

- `task.set_caching_options(False)` disables it for one task.
- `enable_caching` at submission overrides it for a whole run.
- The compiler flag `--disable-execution-caching-by-default`, or the variable `KFP_DISABLE_EXECUTION_CACHING_BY_DEFAULT`, changes the default.

KFP 2.17 adds an optional [MLflow plugin](https://www.kubeflow.org/docs/components/pipelines/operator-guides/mlflow-plugin/). When an administrator enables it:

- The API server creates an MLflow parent run for each pipeline run. It tags the parent with `kfp.pipeline_run_id` and `kfp.pipeline_run_url`, and with `kfp.pipeline_id` and `kfp.pipeline_version_id` where they exist.
- Each task gets a nested MLflow run.
- The MLflow IDs appear in the run's `plugins_output.mlflow`.

The KFP main branch is replacing MLMD with storage in the API server and moving the API to `/apis/v2`. No release up to 2.17.3 includes either change.

### The REST API

The API server exposes the [v2beta1 API](https://github.com/kubeflow/pipelines/tree/2.17.3/backend/api/v2beta1) through grpc-gateway, alongside `/apis/v1beta1`. The read endpoints that matter here:

| Endpoint | Returns |
| --- | --- |
| `GET /apis/v2beta1/runs` | `runs`, `total_size`, `next_page_token` |
| `GET /apis/v2beta1/runs/{run_id}` | One run |
| `GET /apis/v2beta1/pipelines` | `pipelines`, `total_size`, `next_page_token` |
| `GET /apis/v2beta1/pipelines/{pipeline_id}/versions` | `pipeline_versions`, `total_size`, `next_page_token` |
| `GET /apis/v2beta1/recurringruns` | `recurringRuns`, `total_size`, `next_page_token` |
| `GET /apis/v2beta1/experiments` | `experiments`, `total_size`, `next_page_token` |
| `GET /apis/v2beta1/healthz` | `commit_sha`, `tag_name`, `multi_user`, `pipeline_store` |

**JSON.** Field names are snake_case, as in the protos, except `recurringRuns`, which the proto itself spells in camelCase. 64-bit integers such as `interval_second` arrive as strings, enums as their names, and timestamps in RFC 3339. Two timestamps need care:

- `scheduled_at` equals `created_at` for a one-off run.
- An unfinished run reports `finished_at` as `1970-01-01T00:00:00Z`, the Unix epoch, rather than omitting it.

**Paging.** `page_size` defaults to 20 and is capped at 200. A response's `next_page_token` goes back as `page_token` to fetch the next page, and is empty on the last.

**Sorting and filtering.** `sort_by` takes `field` or `field desc`. `filter` takes URL-encoded JSON: a list of predicates, all of which must match. Each predicate has a `key`, a value, and an `operation`: `EQUALS`, `NOT_EQUALS`, `GREATER_THAN`, `GREATER_THAN_EQUALS`, `LESS_THAN`, `LESS_THAN_EQUALS`, `IN`, or `IS_SUBSTRING`.

```shell
$ curl -sG http://ml-pipeline.kubeflow.svc:8888/apis/v2beta1/runs \
    -H "Authorization: Bearer $TOKEN" \
    --data-urlencode namespace=team-ml \
    --data-urlencode page_size=200 \
    --data-urlencode 'sort_by=created_at desc' \
    --data-urlencode 'filter={"predicates":[{"key":"state","operation":"EQUALS","string_value":"RUNNING"}]}'
```

**Identity.** In multi-user mode the API server takes the first identity it can establish:

- **A header.** The Kubeflow manifests set `USERID_HEADER` to `kubeflow-userid` with an empty `USERID_PREFIX`. KFP's own default is `x-goog-authenticated-user-email` with the prefix `accounts.google.com:`.
- **A bearer token.** A Kubernetes TokenReview validates it, and its audience must include `pipelines.kubeflow.org` (`TOKEN_REVIEW_AUDIENCE`).

**Authorisation.** The API server then runs a SubjectAccessReview for verbs such as `get` and `list` on resources in the `pipelines.kubeflow.org` group:

- `runs`;
- `pipelines`, which also covers pipeline versions;
- `experiments`;
- `jobs`, for recurring runs.

A run listing with neither `namespace` nor `experiment_id` is rejected with "A run cannot have an empty namespace in multi-user mode". Reads of shared pipelines need no authorisation, and with `MULTIUSER_SHARED_READ` every `get` and `list` is allowed. In single-user mode the API server makes no checks at all, and `healthz` needs no authentication in either mode.

**Run states** (`RuntimeState`):

| State | Meaning |
| --- | --- |
| `PENDING` | Created; not yet executing. |
| `RUNNING` | Executing. |
| `SUCCEEDED` | Completed successfully. |
| `SKIPPED` | Not executed, for example because the result came from the cache. |
| `FAILED` | Failed. |
| `CANCELING` | Termination requested; still stopping. |
| `CANCELED` | Terminated. |
| `PAUSED` | Paused. |
| `RUNTIME_STATE_UNSPECIFIED` | Default value; not used. |

## Other Kubeflow components

**[Katib](https://www.kubeflow.org/docs/components/katib/)** tunes hyperparameters through CRDs in `kubeflow.org/v1beta1`:

- An `Experiment` declares an objective (a metric to maximise or minimise, optionally with a goal), a search algorithm (`random`, `grid`, `bayesianoptimization`, `tpe`, `cmaes`, `hyperband`, and others), a parameter space, and a trial template.
- A `Suggestion` proposes parameter assignments.
- Each `Trial` runs the template with one assignment. The template can create a Job, a training job, or an Argo Workflow.
- A metrics collector records each trial's observations.

The Experiment's status holds trial counts and `currentOptimalTrial`. With `katib: true`, the adapter reads Experiments and Trials through the Kubernetes API.

**The Training Operator and [Kubeflow Trainer](https://www.kubeflow.org/docs/components/trainer/)** run distributed training jobs, with different APIs:

- The Training Operator (v1.x) defines framework-specific jobs in `kubeflow.org/v1`: `PyTorchJob`, `TFJob`, `MPIJob`, `XGBoostJob`, `PaddleJob`, and, from v1.9, `JAXJob`.
- Trainer 2.x, shipped from Kubeflow 1.11, defines one `TrainJob` in `trainer.kubeflow.org/v1alpha1`. `TrainingRuntime` and `ClusterTrainingRuntime` resources configure it, and it is built on JobSet.

The adapter reads neither. A pipeline task that submits such a job appears only as that task.

**[KServe](https://github.com/kserve/kserve)** serves models as `InferenceService` resources (`serving.kserve.io/v1beta1`). Each names a model format and a storage URI, and reports a URL and readiness conditions. The adapter does not read it.

**[Model Registry](https://www.kubeflow.org/docs/components/hub/)**, distributed in 26.03 as Kubeflow Hub, stores registered models, versions, and model artifacts behind a REST API (`/api/model_registry/v1alpha3/...`). The adapter does not read it. To bring models into the lattice, use the mlflow adapter's `registry` option.

**[Notebooks](https://www.kubeflow.org/docs/components/notebooks/)** runs `Notebook` resources (`kubeflow.org/v1`) in profile namespaces. Each one becomes a StatefulSet and a Service, routed under `/notebook/<namespace>/<name>/`. With `notebooks: true`, the adapter reads Notebook resources through the Kubernetes API.

## How the kubeflow adapter reads Kubeflow

### At a glance

| | |
| --- | --- |
| Name and display name | `kubeflow`, "Kubeflow" |
| Upstream | The KFP v2beta1 REST API; Katib and Notebooks through the Kubernetes API |
| Facet kinds declared | `pipeline`, `workflow` |
| Correlation keys declared | `kfp.run_id`, `kfp.pipeline_id`, `metaflow.run_id`, `mlflow.run_id`, `kfp.recurring_run_id` |
| Watchable | Yes |
| Maturity | Canonical |
| Required option | `pipelines_url` |
| Writes upstream | Never |

`fendlemark adapters describe kubeflow` prints this descriptor as JSON, including the configuration schema.

### Connecting

`Connect` reads the adapter's options and makes no request:

1. **Base URL.** It requires `pipelines_url`, an absolute URL with a scheme and a host, and removes a trailing slash.
2. **HTTP client.** It builds a client whose per-request timeout is the adapter's `timeout`.
3. **Bearer token.** It takes a token from the adapter's `token` option or, failing that, the `KUBEFLOW_TOKEN` environment variable.
4. **Options.** It records `profiles`, `dashboard_url` (trailing slash removed), and `labels`.

Every request carries `Accept: application/json` and `User-Agent: fendlemark/0.9.12`.

### What it requests

For each profile in `profiles`, the adapter lists three resources with `namespace=<profile>`. When `profiles` is empty, it lists each once with no `namespace` parameter.

| Request | Query | Paging |
| --- | --- | --- |
| `GET /apis/v2beta1/runs` | `page_size=200`, `sort_by=created_at desc`, `namespace` | Follows `next_page_token` to the last page. |
| `GET /apis/v2beta1/pipelines` | `page_size=200`, `namespace` | First page only. |
| `GET /apis/v2beta1/recurringruns` | `page_size=200`, `namespace` | First page only. An error here is ignored and yields no recurring-run facets. |
| `GET /apis/v2beta1/healthz` | none | Health checks only. |

The adapter sends no `filter`. Runs come newest first and include `run_details.task_details`, so a page of runs with many tasks is large.

### When it reads

**Full reconciliation.** At start, the control plane runs a full reconciliation. The adapter lists runs, pipelines, and recurring runs in every profile, and the lattice replaces all kubeflow facets with the result. `serve` reports ready on `/readyz` once this completes.

**The watch.** Because the adapter is watchable, the control plane then holds its watch open for the life of the process. The watch covers runs only:

- It emits an `upsert` event for a run whenever the run's status or finish time differs from what it last emitted. It starts with one event for every run.
- Each event is applied to the lattice at once.
- It reports changes, not deletions.

Pipelines and recurring runs are read at full reconciliation, which is also when a deleted run leaves the lattice. Both take effect when the process next starts.

**One-shot commands.** `fendlemark adapters check`, `fendlemark doctor`, and `fendlemark lattice inspect`, `explain`, and `stats` each connect and run a full reconciliation in their own process.

### Authentication

The adapter supports both identities the API server understands:

- **Bearer token (recommended).** Set `KUBEFLOW_TOKEN`, or the adapter's `token` option, to a Kubernetes service account token whose audience includes `pipelines.kubeflow.org`. The adapter sends it as `Authorization: Bearer <token>`. It reads the value only when it connects, so a token that expires while the process runs is not refreshed. `fendlemark doctor` redacts it in the effective configuration.
- **User header.** When the adapter's `user_namespace` option is set, the adapter also sends `kubeflow-userid` with the value of its `user_id` option. Kubeflow's Istio policy denies a request that carries this header unless it comes from one of KFP's own service accounts. Use the header only where nothing in front of the API server enforces that policy.

In single-user mode no credentials are needed.

### Facets

**Pipeline Run**, one per KFP run:

| Facet attribute | Value |
| --- | --- |
| `source` / `source_id` | `kubeflow` / `run:<run_id>` |
| `kind` | `workflow` |
| `title` | `display_name` |
| `status` | `state`, mapped as in the phase table below |
| `started_at` | `scheduled_at`; if absent, `created_at` |
| `ended_at` | `finished_at`, which is the Unix epoch until the run finishes |
| `keys` | `kfp.run_id` (always); `kfp.pipeline_id` when the run references a pipeline; `mlflow.run_id` and `metaflow.run_id` from parameters |
| `fields.namespace` | The run's `namespace`. The v2beta1 `Run` has no such field, since a run's namespace is its experiment's, so on KFP 2.x it is empty. |
| `fields.experiment_id`, `fields.pipeline_version_id` | Copied from the run |
| `fields.state` | The KFP state, unmapped |
| `fields.parameters` | `runtime_config.parameters` |
| `fields.tasks` | One entry per task detail: `name`, `phase` (mapped), `started_at`, `ended_at` |
| `links` | "Open in Kubeflow": `<dashboard_url>/pipeline/#/runs/details/<run_id>`, when `dashboard_url` is set |

A run parameter named exactly `mlflow.run_id` or `mlflow_run_id` becomes the `mlflow.run_id` key, and one named `metaflow.run_id` or `metaflow_run_id` becomes `metaflow.run_id`. The value must be a non-empty string.

**Pipeline** and **Recurring Run**:

| Facet attribute | Pipeline | Recurring Run |
| --- | --- | --- |
| `source_id` | `pipeline:<pipeline_id>` | `recurring:<recurring_run_id>` |
| `kind` | `pipeline` | `pipeline` |
| `title` | `display_name` | `display_name` |
| `status` | `Succeeded` | `Succeeded` when `status` is `ENABLED`, otherwise `Pending` |
| `started_at` | `created_at` | `created_at` |
| `keys` | `kfp.pipeline_id` | `kfp.recurring_run_id` |
| `fields` | `namespace`, `description`, `type: Pipeline` | `namespace`, `type: RecurringRun`, `status`, `schedule` (the cron string, or `every <n>s` for a periodic trigger) |
| `links` | `<dashboard_url>/pipeline/#/pipelines/details/<pipeline_id>` | `<dashboard_url>/pipeline/#/recurringrun/details/<recurring_run_id>` |

Each facet also carries a `label.<key>` field for every entry in the adapter's `labels`.

Run facets do not carry `kfp.recurring_run_id`, so a run is not joined to the recurring run that started it. A run's pipeline is a separate entity, too: facets sharing a `kfp.pipeline_id` all come from this adapter, so they never merge. Instead, `fendlemark lattice inspect --key kfp.pipeline_id=<id>` lists the pipeline and its runs (up to `--limit`, default 50), because a key query matches any facet of an entity.

**Phase mapping.** The same mapping applies to runs and to tasks:

| KFP state | Fendlemark phase |
| --- | --- |
| `PENDING`, `SKIPPED` | Pending |
| `RUNNING`, `CANCELING` | Running |
| `SUCCEEDED`, `CACHED` | Succeeded |
| `FAILED`, `CANCELED`, `PAUSED` | Failed |
| anything else | Unknown |

A task satisfied from the cache is `SKIPPED`, so cached tasks can appear in `tasks` with phase Pending.

### Katib, Notebooks, experiments, and pipeline versions

Beyond the three facet types above, the adapter covers:

- **KFP.** It reads pipeline versions and KFP experiments in the listed profiles, and contributes KFP Experiment facets.
- **Katib.** With `katib: true` (the default), it reads Katib `Experiment` and `Trial` resources through the Kubernetes API. It contributes Katib Experiment facets: objective, algorithm, and a summary of the best trial.
- **Notebooks.** With `notebooks: true` (the default), it reads `Notebook` resources the same way and contributes Notebook Server facets: image, resources, and status. The console's Notebooks view lists them by profile.

Kubernetes access follows the standard `KUBECONFIG` conventions.

### Correlation keys

| Key | Where the adapter gets it | What it joins |
| --- | --- | --- |
| `kfp.run_id` | The run ID, on every run facet | The Argo Workflow, whose `pipeline/runid` label the argo adapter reads as `kfp.run_id`; an MLflow run tagged `kfp.run_id` |
| `kfp.pipeline_id` | `pipeline_version_reference.pipeline_id`; the pipeline's ID | Nothing across adapters out of the box |
| `argo.workflow.name` | Not provided. KFP names the Workflow from the pipeline name with a generated suffix, and the run record does not hold it. | Nothing; the Argo join uses `kfp.run_id` |
| `mlflow.run_id` | A run parameter named `mlflow.run_id` or `mlflow_run_id` | The MLflow run |
| `metaflow.run_id` | A run parameter named `metaflow.run_id` or `metaflow_run_id` | A Metaflow run, and MLflow runs tagged with it |
| `kfp.recurring_run_id` | The recurring run's ID | Nothing across adapters |

The join between a KFP run and its Workflow is automatic: KFP labels every Workflow it creates, and the argo adapter turns that label into the same `kfp.run_id` value the kubeflow adapter emits. The MLflow join takes one line in the training code: tag the MLflow run `kfp.run_id`, with the value from `dsl.PIPELINE_JOB_ID_PLACEHOLDER`.

The KFP MLflow plugin does not provide this join. It tags the parent run `kfp.pipeline_run_id`, which the mlflow adapter does not map to `kfp.run_id`. Adding that tag to the mlflow adapter's `correlation_tags` would produce a key also named `kfp.pipeline_run_id`, which no kubeflow facet carries. The kubeflow adapter does not read `plugins_output`.

### How the lattice joins them

The lattice applies these rules (see [Architecture](https://fendlemark.app/docs/architecture/#correlation)):

1. **Grouping.** Facets are grouped by each strong key and value.
2. **Chaining.** Within a group, facets are sorted by `started_at`. Each is linked to its neighbour when the gap is at most `lattice.correlation.window` (default `90s`). Two facets from the same adapter never merge.
3. **Kind.** The entity's kind follows precedence: run, then workflow, then pipeline. A KFP run with its Workflow is a `workflow` entity; add an MLflow run and it is a `run` entity.
4. **Title.** The title comes from the first facet in adapter order: mlflow, metaflow, argo, kubeflow. A KFP run merged with its Workflow therefore shows the Workflow's name.
5. **Status.** Failed if any facet failed; otherwise Running if any is running, in which case the entity has no end time; otherwise Pending if any is pending; Succeeded if all succeeded; and Unknown otherwise.

The KFP run's `started_at` comes from `scheduled_at`, set when the run was created. The Workflow's comes from Argo's `status.startedAt`, normally seconds later. An MLflow run opened by a later task starts after the earlier tasks finish, so its gap can exceed the window. `fendlemark lattice explain` shows every gap.

### Health, maturity, and error codes

**Health.** `Health` calls `GET /apis/v2beta1/healthz` within `timeout`:

- On success the state is `healthy`, with the detail `pipelines API reachable`. The console and `adapters check` append the facet count.
- On failure it is `unreachable`, with the error as detail.
- Before `Connect` it is `disabled`, with the detail `not connected`.

Health is checked on demand: by `GET /api/v1/adapters`, by each `/metrics` scrape (`fendlemark_adapter_up{adapter="kubeflow"}`), by `fendlemark adapters check`, and by `doctor`. KFP serves `healthz` without authentication, so `healthy` proves the API is reachable, not that the adapter may read it.

**States.** A failed `Connect` marks the adapter `unreachable`, and the control plane retries with backoff. A failed listing during full reconciliation marks it `degraded`, with the error as detail.

**Maturity.** The adapter is Canonical.

**Configuration codes:**

| Code | When |
| --- | --- |
| `DOC-111` | `adapters.kubeflow.pipelines_url is required when the adapter is enabled` |
| `DOC-113` | An option fails the adapter's schema, for example `adapters.kubeflow: profiles: want array, got a string` or `adapters.kubeflow: katib: want boolean, got a string` |
| `DOC-301` | `lattice.correlation.window` is below `5s` (warning) |

**Runtime errors:**

- `kubeflow: pipelines_url is required`
- `kubeflow: invalid base URL "<url>"`
- `kubeflow: runs: <url>: HTTP <status>: <body>`, with the body cut at 160 characters
- `kubeflow: pipelines: …`, in the same form
- `adapter: not connected`

## Configuration reference

The common keys under `adapters.kubeflow` (see [Configuration](https://fendlemark.app/docs/configuration/#adapters)):

| Key | Type | Default | Description |
| --- | --- | --- | --- |
| `enabled` | bool | `false` | Construct the adapter at start. |
| `poll_interval` | duration | `30s` | For adapters without a watch API. Ignored by argo and kubeflow when watching. |
| `timeout` | duration | `10s` | Per-request timeout against the KFP API. |
| `max_entities` | int | `50000` | Soft cap on facets held from this adapter; oldest dropped first. |
| `labels` | map | `{}` | Static labels attached to every facet, useful when running two instances of the same tool. |

The adapter-specific keys:

| Key | Type | Default | Description |
| --- | --- | --- | --- |
| `pipelines_url` | string | required | KFP API base URL, for example `http://ml-pipeline.kubeflow.svc:8888`. |
| `profiles` | list | all visible | Profiles (namespaces) to read. In multi-user mode, list them: the API rejects run listings without a namespace. |
| `katib` | bool | `true` | Read Katib Experiments through the Kubernetes API. |
| `notebooks` | bool | `true` | Read Notebook servers through the Kubernetes API. |
| `dashboard_url` | string | unset | Central Dashboard URL for deep links. |

Three lattice keys govern how kubeflow facets join others: `lattice.correlation.window` (`90s`), `lattice.retain_orphans` (`1h`), and `lattice.max_entities` (`200000`).

Any key can be overridden from the environment. For example, `FENDLEMARK_ADAPTERS__KUBEFLOW__PROFILES='[team-ml,team-vision]'` sets a list. Any string value may reference `${VAR}`.

```yaml
adapters:
  kubeflow:
    enabled: true
    pipelines_url: "http://ml-pipeline.kubeflow.svc:8888"
    profiles: ["team-ml", "team-vision"]
    dashboard_url: "https://kubeflow.example.internal"
    timeout: 20s
    labels:
      cluster: prod-eu
```

## Permissions and least privilege

**What to grant.** In a multi-user deployment, give Fendlemark its own ServiceAccount, with read access in each profile it lists and nothing else:

- `get` and `list` on KFP's resources;
- if Katib and Notebooks are enabled, `get`, `list`, and `watch` on `experiments.kubeflow.org`, `trials.kubeflow.org`, and `notebooks.kubeflow.org`.

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fendlemark-kfp-read
rules:
- apiGroups: ["pipelines.kubeflow.org"]
  resources: ["runs", "pipelines", "experiments", "jobs"]
  verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: fendlemark-kfp-read
  namespace: team-ml            # one RoleBinding per listed profile
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fendlemark-kfp-read
subjects:
- kind: ServiceAccount
  name: fendlemark
  namespace: fendlemark
```

**Why not an existing role.** A ClusterRole bound through RoleBindings grants nothing outside the bound namespaces. Prefer this one to `kubeflow-pipelines-view`, which also allows `readArtifact` on runs, creating and deleting viewers, and creating visualisations. Making Fendlemark a profile contributor grants more still, because `kubeflow-view` aggregates read access to every Kubeflow component.

**The token.** Issue a token with the audience the API server checks:

```shell
$ kubectl create token fendlemark -n fendlemark --audience pipelines.kubeflow.org --duration 24h
```

Pass it through `KUBEFLOW_TOKEN` or `${VAR}`, and restart before it expires. The cluster may cap `--duration` (`--service-account-max-token-expiration`).

**The network.** Point `pipelines_url` at `ml-pipeline` in the `kubeflow` namespace, and leave `user_namespace` unset. Istio's `ml-pipeline` policy admits requests without the user header, and the API server then checks the token. To go through the ingress gateway instead, the gateway must accept service account tokens through `istio-m2m`, and `pipelines_url` becomes the host followed by `/pipeline`.

**Other modes.** With `MULTIUSER_SHARED_READ`, reads need no RoleBinding. In single-user mode the API server authorises nothing, and the network path is the only control.

## Operating the adapter

### Request volume and scale

A full reconciliation over P profiles makes 2P requests for pipelines and recurring runs, plus one runs request per 200 runs in each profile. Three profiles of 12,000 runs each take 186 requests. The watch reads run changes through the same runs listing.

To keep this in bounds:

- **List only the profiles you use.** Each one adds at least three requests to every full reconciliation.
- **Raise `timeout`** if pages of runs with large task lists are slow. It applies to each request, not to a whole listing.
- **Limit KFP's own run history.** The adapter reads every page of runs, whatever their age.
- **Know what the caps do.** `max_entities` keeps the newest facets at reconciliation, and `lattice.max_entities` caps the whole lattice. Neither reduces requests.
- **Allow for start-up.** The first watch pass emits one event per run, and each event reconciles the lattice. On a large platform, `serve --no-reconcile-on-start` starts serving at once, and the console fills in as the lattice does.

### Failure modes

| Symptom | Cause | Remedy |
| --- | --- | --- |
| `DOC-111` at start | `pipelines_url` missing | Set it. |
| `unreachable`, `invalid base URL` | No scheme or host | Use `http://host:port`. |
| `degraded`, `HTTP 400` on `runs`, about an empty namespace | `profiles` empty on a multi-user API | List the profiles. |
| `degraded`, `HTTP 401` | No token, or one the API server rejected: expired, or not issued for `pipelines.kubeflow.org` | Set `KUBEFLOW_TOKEN`; reissue with `--audience pipelines.kubeflow.org`. |
| `degraded`, `HTTP 403` with a KFP error naming the namespace | No RoleBinding in that profile | Bind the read role there. |
| `degraded`, `HTTP 403`, body `RBAC: access denied` | Istio denied the request, usually because it carried `kubeflow-userid` | Unset `user_namespace`; use the token. |
| `healthy`, but run states stop changing | Token expired or permission revoked; the watch skips failed listings, and `healthz` needs no authentication | Check with `curl` (below); renew the token and restart. |
| Recurring runs missing | Their listing failed, which is ignored, or a profile has more than 200 | Check permission on `jobs`; only the first page is read. |
| Shared pipelines missing | With `profiles` set, every request names a namespace | Expected; shared pipelines are listed only without one. |
| New pipelines missing | Pipelines are read at full reconciliation | They appear at the next start. |
| A KFP run has no Argo facet | The argo adapter does not read the profile namespace, or the Workflow was removed after its TTL | Add the profile to `adapters.argo.namespaces`; older runs keep only their KFP facet. |
| MLflow run not merged | Tag missing, or the gap exceeds the window | Tag `kfp.run_id`; read the gap in `explain`; widen `lattice.correlation.window`. |
| `ended_at` of 1970 on an unfinished run | KFP reports the epoch until a run finishes | Treat it as unset; an entity drops its end time while any facet is running. |
| Finished run with Pending tasks | Tasks satisfied from the cache are `SKIPPED` | Expected. |

### Troubleshooting

Start with what KFP reports. `healthz` shows the version and the mode:

```shell
$ curl -s http://ml-pipeline.kubeflow.svc:8888/apis/v2beta1/healthz
{"commit_sha":"…","tag_name":"2.16.1","multi_user":true,…}
$ curl -s -H "Authorization: Bearer $KUBEFLOW_TOKEN" \
    "http://ml-pipeline.kubeflow.svc:8888/apis/v2beta1/runs?namespace=team-ml&page_size=1"
```

An adapter's detail shows only the first 160 characters of an HTTP error body, so repeat a failing request with `curl` to read all of it.

Check the RoleBinding as the API server will. kubectl may warn that `runs.pipelines.kubeflow.org` is not a served resource; the answer still comes from the authoriser:

```shell
$ kubectl auth can-i list runs.pipelines.kubeflow.org -n team-ml \
    --as system:serviceaccount:fendlemark:fendlemark
```

Then look from Fendlemark's side:

```shell
$ fendlemark adapters check kubeflow        # exit 0 healthy, 2 degraded, 3 unreachable
$ fendlemark lattice stats                  # counts by kind and source, correlations by key
$ fendlemark lattice inspect --key kfp.run_id=<run id>
$ fendlemark lattice explain <entity id>
```

On a running server, `/metrics` shows the same counts as `fendlemark_facets{source="kubeflow"}` and `fendlemark_correlations{key="kfp.run_id"}`. If the second is absent or zero while both the kubeflow and argo adapters hold facets, nothing is merging on `kfp.run_id`.

## Worked example

A team in profile `team-ml` trains a churn model with KFP. Fendlemark runs the kubeflow, argo, and mlflow adapters. The argo adapter reads `team-ml` through an Argo Server deployed separately, since Kubeflow does not install one.

### 1. The pipeline

```python
from kfp import compiler, dsl

@dsl.component(base_image="python:3.12", packages_to_install=["pandas"])
def prepare(rows: int, data: dsl.Output[dsl.Dataset]):
    ...  # writes a training set to data.path

@dsl.component(base_image="python:3.12", packages_to_install=["mlflow", "scikit-learn", "pandas"])
def train(data: dsl.Input[dsl.Dataset], epochs: int, kfp_run_id: str, model: dsl.Output[dsl.Model]):
    import mlflow  # MLFLOW_TRACKING_URI is set in the task's environment
    with mlflow.start_run(run_name="churn-train", tags={"kfp.run_id": kfp_run_id}):
        mlflow.log_param("epochs", epochs)
        ...  # trains, logs metrics, writes model.path

@dsl.pipeline(name="churn-train")
def churn_train(rows: int = 100000, epochs: int = 10):
    prep = prepare(rows=rows)
    train(data=prep.outputs["data"], epochs=epochs, kfp_run_id=dsl.PIPELINE_JOB_ID_PLACEHOLDER)

compiler.Compiler().compile(churn_train, package_path="churn-train.yaml")
```

### 2. Upload and run

From a notebook in `team-ml` with the pipelines token mounted:

```python
from kfp.client import Client, ServiceAccountTokenVolumeCredentials

client = Client(host="http://ml-pipeline-ui.kubeflow",
                credentials=ServiceAccountTokenVolumeCredentials())
p = client.upload_pipeline("churn-train.yaml", pipeline_name="churn-train", namespace="team-ml")
v = client.list_pipeline_versions(pipeline_id=p.pipeline_id).pipeline_versions[0]
exp = client.create_experiment("churn", namespace="team-ml")
client.run_pipeline(experiment_id=exp.experiment_id, job_name="churn-train 2026-10-25",
                    pipeline_id=p.pipeline_id, version_id=v.pipeline_version_id,
                    params={"epochs": 10})
```

### 3. What each tool records

The run is submitted at 02:00:00. The API server records run `5b0e6c1a-8d2f-4c39-9a71-3e4f5d6c7b80` and creates Workflow `churn-train-x7k2q` in `team-ml`, labelled `pipeline/runid=5b0e6c1a-…`. Argo starts it at 02:00:02. `prepare` takes a minute, then `train` opens MLflow run `9e3c7a1b5d2f4e6a8b0c1d2e3f4a5b6c` at 02:01:05, tagged `kfp.run_id=5b0e6c1a-…`. The Workflow finishes at 02:09:41, and the persistence agent reports the run `SUCCEEDED`.

The adapter's listing of `team-ml` returns the run (abridged):

```json
{
  "run_id": "5b0e6c1a-8d2f-4c39-9a71-3e4f5d6c7b80",
  "display_name": "churn-train 2026-10-25",
  "experiment_id": "c41d…",
  "pipeline_version_reference": {"pipeline_id": "a7f2…", "pipeline_version_id": "e90b…"},
  "runtime_config": {"parameters": {"epochs": 10}},
  "created_at": "2026-10-25T02:00:00Z",
  "scheduled_at": "2026-10-25T02:00:00Z",
  "finished_at": "2026-10-25T02:09:41Z",
  "state": "SUCCEEDED",
  "run_details": {"task_details": [{"display_name": "train", "state": "SUCCEEDED", "start_time": "2026-10-25T02:01:03Z", "end_time": "2026-10-25T02:09:30Z"}]}
}
```

It becomes this facet (abridged: system tasks and some keys omitted):

```json
{
  "source": "kubeflow",
  "source_id": "run:5b0e6c1a-8d2f-4c39-9a71-3e4f5d6c7b80",
  "kind": "workflow",
  "title": "churn-train 2026-10-25",
  "status": "Succeeded",
  "started_at": "2026-10-25T02:00:00Z",
  "ended_at": "2026-10-25T02:09:41Z",
  "keys": {"kfp.run_id": "5b0e6c1a-8d2f-4c39-9a71-3e4f5d6c7b80", "kfp.pipeline_id": "a7f2…"},
  "fields": {"experiment_id": "c41d…", "pipeline_version_id": "e90b…", "state": "SUCCEEDED",
             "parameters": {"epochs": 10},
             "tasks": [{"name": "train", "phase": "Succeeded", "started_at": "2026-10-25T02:01:03Z", "ended_at": "2026-10-25T02:09:30Z"}]},
  "links": [{"label": "Open in Kubeflow", "url": "https://kubeflow.example.internal/pipeline/#/runs/details/5b0e6c1a-8d2f-4c39-9a71-3e4f5d6c7b80"}]
}
```

The other two adapters contribute their own facets:

- **argo**: `team-ml/churn-train-x7k2q`, kind `workflow`, started 02:00:02. Its keys include `argo.workflow.name=churn-train-x7k2q` and, from the label, `kfp.run_id=5b0e6c1a-…`.
- **mlflow**: the MLflow run, kind `run`, started 02:01:05, with `mlflow.run_id` and, from the tag, `kfp.run_id`.

### 4. The entity

```shell
$ fendlemark lattice inspect --key kfp.run_id=5b0e6c1a-8d2f-4c39-9a71-3e4f5d6c7b80
ID                   KIND  STATUS     FACETS                STARTED               TITLE
lt_3d9a61f0c2b84e57  run   Succeeded  argo+kubeflow+mlflow  2026-10-25T02:00:00Z  churn-train
$ fendlemark lattice explain lt_3d9a61f0c2b84e57
lt_3d9a61f0c2b84e57  run  Succeeded  "churn-train"
  facet  mlflow     9e3c7a1b5d2f4e6a8b0c1d2e3f4a5b6c
  facet  argo       team-ml/churn-train-x7k2q
  facet  kubeflow   run:5b0e6c1a-8d2f-4c39-9a71-3e4f5d6c7b80
  merge  kfp.run_id = "5b0e6c1a-8d2f-4c39-9a71-3e4f5d6c7b80" linked kubeflow/run:5b0e6c1a-8d2f-4c39-9a71-3e4f5d6c7b80 and argo/team-ml/churn-train-x7k2q (gap 2s)
  merge  kfp.run_id = "5b0e6c1a-8d2f-4c39-9a71-3e4f5d6c7b80" linked argo/team-ml/churn-train-x7k2q and mlflow/9e3c7a1b5d2f4e6a8b0c1d2e3f4a5b6c (gap 1m3s)
```

Both merges come from `kfp.run_id`: the first from the label KFP set, the second from the tag the training code set. The entity is a `run`, because the MLflow facet outranks the two `workflow` facets, and it takes the MLflow run's name as its title. It runs from 02:00:00, its earliest facet's start, to 02:09:41, its latest facet's end.

The second gap, 63 seconds, is within the 90-second window. If `prepare` grew to three minutes, the MLflow run would stand alone and `explain` would show one merge. To prevent that, widen `lattice.correlation.window` beyond the longest such delay, or open the MLflow run in the first task.

In the console, the entity appears in the Runs view, where its detail page shows each facet with a deep link into its tool. Without the mlflow adapter, the Argo and KFP facets form a `workflow` entity titled `churn-train-x7k2q`, shown in the Workflows view. The pipeline `churn-train` is its own `pipeline` entity in the Pipelines view, and `lattice inspect --key kfp.pipeline_id=a7f2…` lists it with its runs.

## Glossary

| Kubeflow term | Fendlemark term |
| --- | --- |
| Profile | A namespace listed in `profiles`; `fields.namespace` on pipeline and recurring-run facets |
| Pipeline | Pipeline facet: `kind: pipeline`, `source_id` `pipeline:<id>`, key `kfp.pipeline_id` |
| Pipeline version | `fields.pipeline_version_id` on a run facet |
| Experiment (KFP) | `fields.experiment_id` on a run facet |
| Run | Pipeline Run facet: `kind: workflow`, `source_id` `run:<id>`, key `kfp.run_id` |
| Recurring run | Recurring Run facet: `kind: pipeline`, `source_id` `recurring:<id>`, key `kfp.recurring_run_id` |
| Run parameter | `fields.parameters`; `mlflow.run_id` and `metaflow.run_id` parameters become keys |
| Task | An entry in `fields.tasks` |
| Runtime state | `fields.state`, mapped to the facet's phase |
| Argo Workflow of a run | The argo adapter's facet, merged on `kfp.run_id` |
| `pipeline/runid` label | The source of `kfp.run_id` on the argo facet |
| MLMD artifact, execution, context | Not read |
| Katib Experiment | Katib Experiment facet |
| Notebook | Notebook Server facet, kind `notebook` |
| Central Dashboard | `dashboard_url`, the base of every deep link |