# Running the Fendlemark app on Kubernetes Source: https://fendlemark.app/blog/running-the-fendlemark-app-on-kubernetes/ By [the Fendlemark Keepers](https://fendlemark.app/governance/#keepers) · 29 October 2026 The Fendlemark app is one binary: the control plane and the Fendlemark console it serves, one console for your whole ML platform. On a laptop it listens on `127.0.0.1:7411`, and nothing else can reach it. To share it with a team, run it on Kubernetes next to the tools it reads, behind the single sign-on the team already uses. This guide does that without changing the listen address: oauth2-proxy runs in the same pod and signs people in, and every adapter gets credentials that can only read. By the end, the console will be at `https://platform.example.internal`, reachable only through oauth2-proxy, with all four adapters connected. The examples assume an MLflow tracking server at `http://mlflow.mlflow.svc:5000`, a Metaflow metadata service, an Argo Server in the `argo` namespace that runs Workflows in `ml-training` and `ml-batch`, and Kubeflow Pipelines with two profiles, `team-ml` and `team-risk`. Your identity provider can be anything that speaks OpenID Connect. Substitute your own names throughout. ## 1. Keep the app on loopback, with the proxy beside it [`server.listen`](https://fendlemark.app/docs/configuration/#server) defaults to `127.0.0.1:7411`, and the configuration reference advises keeping it there, behind an authenticating reverse proxy, rather than widening it. In a Kubernetes pod the advice is easy to follow, because the containers in a pod share one network namespace. A proxy in the same pod reaches the app on loopback, and no other pod can reach the app directly. ```text browser ──HTTPS──▶ Ingress ──▶ Service :80 ──▶ pod ├─ oauth2-proxy 0.0.0.0:4180 signs people in └─ fendlemark 127.0.0.1:7411 the console and API ``` oauth2-proxy sends each visitor to your identity provider, checks the result, and passes signed-in requests on to `http://127.0.0.1:7411`. `fendlemark doctor` will still report `DOC-201`, because `auth.mode` is `none` and "anyone who can reach server.listen can use the console". In this pod that is intended: the only thing that can reach `server.listen` is the proxy. The one exception is `kubectl port-forward`, which reaches loopback inside the pod, so grant `pods/portforward` in the `fendlemark` namespace as carefully as access to the console itself. ## 2. Create the credentials Every Fendlemark adapter is read-only, and the [adapters reference](https://fendlemark.app/docs/adapters/) lists the least permission each one needs. Create a namespace and a service account for the app, and grant that account no more than that. ```shell $ kubectl create namespace fendlemark $ kubectl -n fendlemark create serviceaccount fendlemark ``` ### MLflow The mlflow adapter reads the same variables as the MLflow client. On a tracking server that uses MLflow's basic authentication, make a user with `READ` on the experiments and registered models you want shown, and store its credentials under keys named exactly like the variables, so the Deployment can import them with `envFrom`: ```shell $ kubectl -n fendlemark create secret generic fendlemark-mlflow \ --from-literal=MLFLOW_TRACKING_USERNAME=fendlemark \ --from-literal=MLFLOW_TRACKING_PASSWORD="$MLFLOW_PASSWORD" ``` Per-experiment permissions narrow anything only if the server's `default_permission` is `NO_PERMISSIONS`; its default is `READ`. For a tracking server that takes bearer tokens, store `MLFLOW_TRACKING_TOKEN` instead. ### Metaflow The Metaflow metadata service has no authentication of its own, so the metaflow adapter needs only network access to it. If you put a proxy in front of the service, give the app a read-only identity there. ### Argo Workflows The argo adapter calls Argo Server with a bearer token. With Argo Server in `client` mode, its default, each request runs with the token's own permissions, so the app sees exactly what this Role allows: `get`, `list`, and `watch` on Workflows, WorkflowTemplates, and CronWorkflows. ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: fendlemark-argo-read namespace: ml-training # one Role and RoleBinding per namespace the adapter reads rules: - apiGroups: ["argoproj.io"] resources: ["workflows", "workflowtemplates", "cronworkflows"] verbs: ["get", "list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: fendlemark-argo-read namespace: ml-training roleRef: {apiGroup: rbac.authorization.k8s.io, kind: Role, name: fendlemark-argo-read} subjects: - {kind: ServiceAccount, name: fendlemark, namespace: fendlemark} --- apiVersion: v1 kind: Secret metadata: name: fendlemark-argo namespace: fendlemark annotations: kubernetes.io/service-account.name: fendlemark type: kubernetes.io/service-account-token ``` Kubernetes fills the Secret's `token` key with a non-expiring token for the `fendlemark` service account. Since Kubernetes 1.24 it creates such a token only when a Secret like this one asks for it. Create the Role and RoleBinding in every namespace you will list under `adapters.argo.namespaces` in step 3, including your Kubeflow profiles. Kubeflow Pipelines runs each pipeline as an Argo Workflow in the profile's namespace, and the argo adapter can show only what Argo Server lets it list. Kubeflow does not install Argo Server, only the workflow controller, so this assumes you have installed one separately and that it can see those namespaces. By default Argo Server serves a self-signed certificate, which the adapter rejects; settle that before step 3, as the first item under [Things that will happen eventually](https://fendlemark.app/blog/running-the-fendlemark-app-on-kubernetes/#gotchas) describes. ### Kubeflow Pipelines In multi-user mode, the Kubeflow Pipelines API server accepts a service account token whose audience includes `pipelines.kubeflow.org`, and authorises it with Kubernetes RBAC in each profile. Define read access to the Pipelines resources once, as a ClusterRole, and bind it with a RoleBinding in each profile so that it grants nothing outside them: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: fendlemark-kfp-read rules: - apiGroups: ["pipelines.kubeflow.org"] resources: ["runs", "pipelines", "experiments", "jobs"] verbs: ["get", "list"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: fendlemark-kfp-read namespace: team-ml # one RoleBinding per profile roleRef: {apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: fendlemark-kfp-read} subjects: - {kind: ServiceAccount, name: fendlemark, namespace: fendlemark} ``` This is narrower than Kubeflow's own `kubeflow-pipelines-view` role, which also lets its holder read run artifacts and logs. Then issue a token with that audience and store it: ```shell $ kubectl -n fendlemark create secret generic fendlemark-kubeflow \ --from-literal=KUBEFLOW_TOKEN="$(kubectl -n fendlemark create token fendlemark \ --audience pipelines.kubeflow.org --duration 168h)" \ --dry-run=client -o yaml | kubectl apply -f - ``` Unlike the Argo token, this one expires, and the kubeflow adapter reads `KUBEFLOW_TOKEN` once, when `serve` starts. Rerun the command above well within the token's 168 hours, followed by `kubectl -n fendlemark rollout restart deployment/fendlemark`. The API server can cap how long a requested token lasts (`--service-account-max-token-expiration`), so check the expiry of the token you received rather than the duration you asked for. A full Kubeflow installation also has a NetworkPolicy, `ml-pipeline` in the `kubeflow` namespace, that admits traffic to the Pipelines API only from profile namespaces and `istio-system`. Requests from the app's namespace are dropped, and since the adapter makes no request when it connects, the log still says `connected`. Add a policy that admits the app's namespace: ```yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: ml-pipeline-from-fendlemark namespace: kubeflow spec: podSelector: matchLabels: app: ml-pipeline policyTypes: ["Ingress"] ingress: - from: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: fendlemark ports: - port: 8888 ``` The kubeflow adapter's `katib` and `notebooks` options default to `true`. Turn them off, as the configuration in step 3 does, unless you want Katib experiments and Notebook servers in the console. Both are read through the Kubernetes API rather than the Pipelines API, so they need Kubernetes credentials found the standard `KUBECONFIG` way, which this Deployment does not mount, plus `get`, `list`, and `watch` on `experiments.kubeflow.org`, `trials.kubeflow.org`, and `notebooks.kubeflow.org`. The [adapters reference](https://fendlemark.app/docs/adapters/#kubeflow) has the details. ### Single sign-on Register an OpenID Connect client for the console with the redirect URI `https://platform.example.internal/oauth2/callback`. Then store the client id, the client secret, and a random cookie secret for oauth2-proxy, which reads each of its flags from an environment variable named `OAUTH2_PROXY_` followed by the flag's name in capitals, with underscores for hyphens: ```shell $ kubectl -n fendlemark create secret generic oauth2-proxy \ --from-literal=OAUTH2_PROXY_CLIENT_ID=fendlemark-console \ --from-literal=OAUTH2_PROXY_CLIENT_SECRET="$OIDC_CLIENT_SECRET" \ --from-literal=OAUTH2_PROXY_COOKIE_SECRET="$(openssl rand -base64 32 | tr -- '+/' '-_')" ``` ## 3. Write the configuration The app reads one file. Keep it in a ConfigMap; the secrets stay in the environment, under names the adapters already read. ```yaml apiVersion: v1 kind: ConfigMap metadata: name: fendlemark-config namespace: fendlemark data: fendlemark.yaml: | server: base_url: "https://platform.example.internal" data_dir: "/var/lib/fendlemark" console: title: "ML Platform" default_view: runs adapters: mlflow: enabled: true tracking_uri: "http://mlflow.mlflow.svc:5000" metaflow: enabled: true metadata_url: "http://metaflow-metadata.metaflow.svc:8080" ui_url: "https://metaflow-ui.example.internal" argo: enabled: true server: "https://argo-server.argo.svc:2746" namespaces: ["ml-training", "ml-batch", "team-ml", "team-risk"] kubeflow: enabled: true pipelines_url: "http://ml-pipeline.kubeflow.svc:8888" profiles: ["team-ml", "team-risk"] dashboard_url: "https://kubeflow.example.internal" katib: false notebooks: false ``` Three things are deliberate: - **There is no `server.listen`.** The default, loopback on port 7411, is what the proxy expects. - **`server.base_url` is the public address.** The console builds its own links from it, which would otherwise point at the loopback address. - **The profiles are listed.** In multi-user mode, the Pipelines API rejects a run listing that names no namespace, so `profiles` must say which ones to read. Unknown keys are errors, not warnings: a misspelt key under `server`, `console`, or `lattice` makes `serve` exit with code 1 at start-up instead of being ignored. Inside an adapter's block, a missing required option or a value of the wrong type stops it the same way, but a misspelt option name is not caught: `katlb: false` leaves Katib on. Check the effective configuration that `fendlemark doctor` prints. ## 4. Deploy the app and the proxy One Deployment runs both containers. The app takes its configuration from the ConfigMap and its credentials from the Secrets in step 2; the proxy takes the OpenID Connect client. ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: fendlemark namespace: fendlemark spec: replicas: 1 selector: matchLabels: app.kubernetes.io/name: fendlemark template: metadata: labels: app.kubernetes.io/name: fendlemark spec: serviceAccountName: fendlemark automountServiceAccountToken: false # every credential arrives through the environment securityContext: runAsNonRoot: true runAsUser: 65532 runAsGroup: 65532 seccompProfile: type: RuntimeDefault containers: - name: fendlemark image: ghcr.io/tophergopher/fendlemark:0.9.12 args: ["serve", "--config", "/etc/fendlemark/fendlemark.yaml"] envFrom: - secretRef: name: fendlemark-mlflow - secretRef: name: fendlemark-kubeflow env: - name: ARGO_TOKEN valueFrom: secretKeyRef: name: fendlemark-argo key: token readinessProbe: # through the proxy, which passes this path on httpGet: path: /readyz port: 4180 periodSeconds: 10 resources: requests: cpu: 100m memory: 256Mi securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: drop: ["ALL"] volumeMounts: - name: config mountPath: /etc/fendlemark readOnly: true - name: data mountPath: /var/lib/fendlemark - name: oauth2-proxy image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.5 args: - --provider=oidc - --oidc-issuer-url=https://sso.example.internal/realms/platform - --redirect-url=https://platform.example.internal/oauth2/callback - --email-domain=example.com - --upstream=http://127.0.0.1:7411/ - --http-address=0.0.0.0:4180 - --reverse-proxy=true - --trusted-proxy-ip=10.244.0.0/16 # your cluster's pod CIDR; step 5 admits only the ingress controller - --skip-provider-button=true - "--skip-auth-route=GET=^/(healthz|readyz)$" envFrom: - secretRef: name: oauth2-proxy ports: - name: http containerPort: 4180 livenessProbe: httpGet: path: /ping port: http securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: drop: ["ALL"] volumes: - name: config configMap: name: fendlemark-config - name: data emptyDir: {} ``` The parts that are not boilerplate: - **The image is the binary and nothing else.** `ghcr.io/tophergopher/fendlemark` is built `FROM scratch` for `linux/amd64`, `linux/arm64`, and `linux/arm/v7`, and runs as user 65532. It has no shell and writes only to `server.data_dir`, so `readOnlyRootFilesystem` does not get in its way. The same image is on Docker Hub as `tophergopher/fendlemark`. Pin a version, as here, rather than `latest`, so that a restart to pick up a new Kubeflow token does not also upgrade the console. - **The data directory is an `emptyDir`.** With `auth.mode: none`, the only state in `server.data_dir` is console preferences, which reset when the pod is replaced. Respiratory Identity would keep its enrolments there too. - **Readiness goes through the proxy.** The kubelet probes the pod's IP address, not its loopback, so it cannot reach port 7411 directly. The `--skip-auth-route` flag lets `GET /healthz` and `GET /readyz` through to the app without signing in. `serve` does not listen at all until its initial reconciliation has finished, so until then the proxy answers `502`, the probe fails, and the pod receives no traffic. On a large platform, nobody sees a half-built console. - **The app has no liveness probe.** `/healthz` only says the process is up, and if the process exits, the kubelet restarts the container anyway. A liveness probe routed through the proxy would restart the app whenever the proxy failed, though the app itself was fine. - **`--reverse-proxy` trusts forwarded headers.** With it, oauth2-proxy believes the `X-Forwarded-*` headers it is sent. `--trusted-proxy-ip` limits that trust to pod addresses, and the NetworkPolicy in step 5 narrows those to the ingress controller's pods; if your controller uses host networking, list the nodes' addresses instead. Without the flag, oauth2-proxy warns at start-up and trusts every source. - **One replica.** Two replicas would work: the lattice is deterministic, so once both have read the same facets they show the same entities with the same ids, and oauth2-proxy keeps its sessions in a cookie, so either can serve a signed-in person. Each counts its own lattice generations, though, so the number in the footer differs between them, and each reads every upstream independently, which doubles the load on MLflow, the Metaflow metadata service, Argo Server, and the Pipelines API. Run one unless the console's availability matters more than that load. ## 5. Expose the proxy through an Ingress The Service points at the proxy's port, never at 7411, and a NetworkPolicy keeps everything but the ingress controller away from it: ```yaml apiVersion: v1 kind: Service metadata: name: fendlemark namespace: fendlemark spec: selector: app.kubernetes.io/name: fendlemark ports: - name: http port: 80 targetPort: http --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: fendlemark namespace: fendlemark spec: ingressClassName: internal # your ingress controller's class tls: - hosts: ["platform.example.internal"] secretName: platform-tls rules: - host: platform.example.internal http: paths: - path: / pathType: Prefix backend: service: name: fendlemark port: name: http --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: fendlemark-ingress namespace: fendlemark spec: podSelector: matchLabels: app.kubernetes.io/name: fendlemark policyTypes: ["Ingress"] ingress: - from: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: ingress # your ingress controller's namespace ports: - port: 4180 ``` With the NetworkPolicy, other pods in the cluster cannot reach the proxy except through the Ingress and its TLS. Use your own ingress controller's class and namespace; Kubernetes labels every namespace with `kubernetes.io/metadata.name`. A NetworkPolicy takes effect only if your cluster's network plugin enforces it. ## 6. Check it Wait for the rollout, then check that each adapter connected and the first reconciliation finished: ```shell $ kubectl -n fendlemark rollout status deployment/fendlemark deployment "fendlemark" successfully rolled out $ kubectl -n fendlemark logs deployment/fendlemark -c fendlemark | grep -E 'adapter|lattice|reconciliation' time=2026-10-29T10:02:11.204Z level=INFO msg="fendlemark starting" version=0.9.12 adapters=argo,kubeflow,metaflow,mlflow time=2026-10-29T10:02:11.204Z level=INFO msg="adapter connected" adapter=argo took=0s time=2026-10-29T10:02:11.204Z level=INFO msg="adapter connected" adapter=kubeflow took=0s time=2026-10-29T10:02:11.204Z level=INFO msg="adapter connected" adapter=metaflow took=0s time=2026-10-29T10:02:11.204Z level=INFO msg="adapter connected" adapter=mlflow took=0s time=2026-10-29T10:02:14.117Z level=INFO msg="lattice reconciled" entities=6112 facets=7986 generation=1 took=97ms time=2026-10-29T10:02:14.117Z level=INFO msg="adapter watching" adapter=argo time=2026-10-29T10:02:14.117Z level=INFO msg="adapter watching" adapter=kubeflow ``` "Connected" means only that an adapter read its configuration; it makes no request until the first reconciliation. An adapter that cannot list, for a missing permission, a rejected token, or a certificate it does not trust, is named in a `level=WARN` line, `msg="initial reconciliation had errors"`, which the `grep` above also shows. A rollout that never finishes is almost always a pod that never becomes ready. To see what the probe sees, forward the proxy's port and ask: ```shell $ kubectl -n fendlemark port-forward deployment/fendlemark 4180:4180 & $ curl -s -o /dev/null -w '%{http_code}\n' http://localhost:4180/readyz 502 ``` A `502` means the app is not listening yet: either the initial reconciliation is still running, or the container has exited, and its logs will say which. A `200` means the pod is ready. Then open `https://platform.example.internal` and sign in. The Adapters view shows each adapter's health, latency, facet counts, and effective configuration, from the same checks that `fendlemark adapters check` and `GET /api/v1/adapters` run. ## Things that will happen eventually - **Argo Server's certificate is self-signed.** Argo Server encrypts by default with a self-signed certificate, and the argo adapter always verifies certificates, so a default installation leaves the adapter `unreachable` with a certificate error. Either give Argo Server a certificate from a CA you control and point `SSL_CERT_FILE` at a bundle mounted in the pod, or, if the traffic never leaves the cluster, run Argo Server with `--secure=false` and set `server` to its `http://` address. - **A CA bundle replaces the built-in roots.** The image has no certificate store, and the binary uses the root certificates it carries only when it finds no others. A bundle in `SSL_CERT_FILE` therefore replaces them, so include public CAs in it if anything the app talks to needs one. - **"Open in MLflow" and "Open in Argo" links point at the cluster.** The console builds them from `tracking_uri` and `server`, so in-cluster addresses like the ones above do not open in a browser. Use addresses that resolve both in the cluster and on people's desks. The Kubeflow and Metaflow links come from `dashboard_url` and `ui_url`, which are already public addresses. - **The Kubeflow token expires quietly.** The kubeflow adapter's watch skips listings that fail, and the Pipelines API's health check needs no token, so the adapter stays `healthy` while Kubeflow run states stop changing. A restart does not show it either: the adapter's health still comes from that check, so it still reports `healthy`, and only the start-up warning that the initial reconciliation had errors names the `HTTP 401`. Reissue the token before it expires, not after; a token issued without `--audience pipelines.kubeflow.org` fails the same way. - **Somebody widens the listen address.** Setting `server.listen` (or `--listen`) to `:7411` lets anything that can reach the pod's IP address skip the proxy, and the app reports `DOC-121` for exactly that reason. - **Metrics are on loopback too.** `/metrics` answers on the same address as the console, so Prometheus cannot scrape it in this layout. Adding it to `--skip-auth-route` makes it scrapable, and also publishes it, unauthenticated, through the Ingress; Prometheus also needs a rule in step 5's NetworkPolicy. Do that only if the metrics may be read by anyone who can reach the console's address. - **Signing in fails at the callback with a `502`.** The session cookie oauth2-proxy sets after an OpenID Connect sign-in can be larger than an ingress controller's buffer for response headers. Raise that buffer; on controllers built on nginx it is `proxy-buffer-size`, and oauth2-proxy's documentation suggests 16k or more. - **The redirect URL must match exactly.** If signing in loops, or your identity provider reports an invalid redirect, compare `--redirect-url` with the URI registered for the client, character by character, including the scheme. ## The console's own sign-in Fendlemark does not integrate with identity providers. Its own sign-in, [Respiratory Identity](https://fendlemark.app/docs/auth/), is off by default: it needs a webcam at every desk and an enrolment for every person, three breaths each. Behind oauth2-proxy, the console admits whoever your identity provider admits and keeps them signed in for as long as the proxy's cookie lasts (`--cookie-expire`, a week by default). If you enable Respiratory Identity anyway, mount a PersistentVolumeClaim at `server.data_dir`, where the enrolments are kept. ## Further reading - [Getting started: the container image](https://fendlemark.app/docs/getting-started/#container), for running the same image on one machine - [Configuration: server](https://fendlemark.app/docs/configuration/#server), for `server.listen` and `DOC-121`, and the rest of `fendlemark.yaml` - [Adapters](https://fendlemark.app/docs/adapters/), for what each adapter reads and the least permission it needs - [CLI reference: `fendlemark doctor`](https://fendlemark.app/docs/cli/#doctor), for `DOC-201` - [Tracking Kubeflow Pipelines runs in MLflow](https://fendlemark.app/blog/tracking-kubeflow-pipelines-runs-in-mlflow/) and [Running MLflow and Argo Workflows together](https://fendlemark.app/blog/running-mlflow-and-argo-workflows-together/), for what to tag so that the console has something to correlate - [oauth2-proxy: configuration overview](https://oauth2-proxy.github.io/oauth2-proxy/configuration/overview), [Argo Workflows: TLS](https://argo-workflows.readthedocs.io/en/latest/tls/), and [Kubernetes: pod networking](https://kubernetes.io/docs/concepts/workloads/pods/#pod-networking)