Fendlemark v0.9.12

How-to

Running the Fendlemark app on Kubernetes

By the Fendlemark Keepers ·

The Fendlemark app is one binary: the control plane and the Fendlemark console it serves, one console for your whole ML platform. On a laptop it listens on 127.0.0.1:7411, and nothing else can reach it. To share it with a team, run it on Kubernetes next to the tools it reads, behind the single sign-on the team already uses. This guide does that without changing the listen address: oauth2-proxy runs in the same pod and signs people in, and every adapter gets credentials that can only read.

By the end, the console will be at https://platform.example.internal, reachable only through oauth2-proxy, with all four adapters connected. The examples assume an MLflow tracking server at http://mlflow.mlflow.svc:5000, a Metaflow metadata service, an Argo Server in the argo namespace that runs Workflows in ml-training and ml-batch, and Kubeflow Pipelines with two profiles, team-ml and team-risk. Your identity provider can be anything that speaks OpenID Connect. Substitute your own names throughout.

1. Keep the app on loopback, with the proxy beside it

server.listen defaults to 127.0.0.1:7411, and the configuration reference advises keeping it there, behind an authenticating reverse proxy, rather than widening it. In a Kubernetes pod the advice is easy to follow, because the containers in a pod share one network namespace. A proxy in the same pod reaches the app on loopback, and no other pod can reach the app directly.

browser ──HTTPS──▶ Ingress ──▶ Service :80 ──▶ pod
                                               ├─ oauth2-proxy    0.0.0.0:4180   signs people in
                                               └─ fendlemark      127.0.0.1:7411 the console and API

oauth2-proxy sends each visitor to your identity provider, checks the result, and passes signed-in requests on to http://127.0.0.1:7411. fendlemark doctor will still report DOC-201, because auth.mode is none and "anyone who can reach server.listen can use the console". In this pod that is intended: the only thing that can reach server.listen is the proxy. The one exception is kubectl port-forward, which reaches loopback inside the pod, so grant pods/portforward in the fendlemark namespace as carefully as access to the console itself.

2. Create the credentials

Every Fendlemark adapter is read-only, and the adapters reference lists the least permission each one needs. Create a namespace and a service account for the app, and grant that account no more than that.

$ kubectl create namespace fendlemark
$ kubectl -n fendlemark create serviceaccount fendlemark

MLflow

The mlflow adapter reads the same variables as the MLflow client. On a tracking server that uses MLflow's basic authentication, make a user with READ on the experiments and registered models you want shown, and store its credentials under keys named exactly like the variables, so the Deployment can import them with envFrom:

$ kubectl -n fendlemark create secret generic fendlemark-mlflow \
    --from-literal=MLFLOW_TRACKING_USERNAME=fendlemark \
    --from-literal=MLFLOW_TRACKING_PASSWORD="$MLFLOW_PASSWORD"

Per-experiment permissions narrow anything only if the server's default_permission is NO_PERMISSIONS; its default is READ. For a tracking server that takes bearer tokens, store MLFLOW_TRACKING_TOKEN instead.

Metaflow

The Metaflow metadata service has no authentication of its own, so the metaflow adapter needs only network access to it. If you put a proxy in front of the service, give the app a read-only identity there.

Argo Workflows

The argo adapter calls Argo Server with a bearer token. With Argo Server in client mode, its default, each request runs with the token's own permissions, so the app sees exactly what this Role allows: get, list, and watch on Workflows, WorkflowTemplates, and CronWorkflows.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: fendlemark-argo-read
  namespace: ml-training        # one Role and RoleBinding per namespace the adapter reads
rules:
  - apiGroups: ["argoproj.io"]
    resources: ["workflows", "workflowtemplates", "cronworkflows"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: fendlemark-argo-read
  namespace: ml-training
roleRef: {apiGroup: rbac.authorization.k8s.io, kind: Role, name: fendlemark-argo-read}
subjects:
  - {kind: ServiceAccount, name: fendlemark, namespace: fendlemark}
---
apiVersion: v1
kind: Secret
metadata:
  name: fendlemark-argo
  namespace: fendlemark
  annotations:
    kubernetes.io/service-account.name: fendlemark
type: kubernetes.io/service-account-token

Kubernetes fills the Secret's token key with a non-expiring token for the fendlemark service account. Since Kubernetes 1.24 it creates such a token only when a Secret like this one asks for it.

Create the Role and RoleBinding in every namespace you will list under adapters.argo.namespaces in step 3, including your Kubeflow profiles. Kubeflow Pipelines runs each pipeline as an Argo Workflow in the profile's namespace, and the argo adapter can show only what Argo Server lets it list. Kubeflow does not install Argo Server, only the workflow controller, so this assumes you have installed one separately and that it can see those namespaces. By default Argo Server serves a self-signed certificate, which the adapter rejects; settle that before step 3, as the first item under Things that will happen eventually describes.

Kubeflow Pipelines

In multi-user mode, the Kubeflow Pipelines API server accepts a service account token whose audience includes pipelines.kubeflow.org, and authorises it with Kubernetes RBAC in each profile. Define read access to the Pipelines resources once, as a ClusterRole, and bind it with a RoleBinding in each profile so that it grants nothing outside them:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fendlemark-kfp-read
rules:
  - apiGroups: ["pipelines.kubeflow.org"]
    resources: ["runs", "pipelines", "experiments", "jobs"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: fendlemark-kfp-read
  namespace: team-ml            # one RoleBinding per profile
roleRef: {apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: fendlemark-kfp-read}
subjects:
  - {kind: ServiceAccount, name: fendlemark, namespace: fendlemark}

This is narrower than Kubeflow's own kubeflow-pipelines-view role, which also lets its holder read run artifacts and logs.

Then issue a token with that audience and store it:

$ kubectl -n fendlemark create secret generic fendlemark-kubeflow \
    --from-literal=KUBEFLOW_TOKEN="$(kubectl -n fendlemark create token fendlemark \
        --audience pipelines.kubeflow.org --duration 168h)" \
    --dry-run=client -o yaml | kubectl apply -f -

Unlike the Argo token, this one expires, and the kubeflow adapter reads KUBEFLOW_TOKEN once, when serve starts. Rerun the command above well within the token's 168 hours, followed by kubectl -n fendlemark rollout restart deployment/fendlemark. The API server can cap how long a requested token lasts (--service-account-max-token-expiration), so check the expiry of the token you received rather than the duration you asked for.

A full Kubeflow installation also has a NetworkPolicy, ml-pipeline in the kubeflow namespace, that admits traffic to the Pipelines API only from profile namespaces and istio-system. Requests from the app's namespace are dropped, and since the adapter makes no request when it connects, the log still says connected. Add a policy that admits the app's namespace:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: ml-pipeline-from-fendlemark
  namespace: kubeflow
spec:
  podSelector:
    matchLabels:
      app: ml-pipeline
  policyTypes: ["Ingress"]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: fendlemark
      ports:
        - port: 8888

The kubeflow adapter's katib and notebooks options default to true. Turn them off, as the configuration in step 3 does, unless you want Katib experiments and Notebook servers in the console. Both are read through the Kubernetes API rather than the Pipelines API, so they need Kubernetes credentials found the standard KUBECONFIG way, which this Deployment does not mount, plus get, list, and watch on experiments.kubeflow.org, trials.kubeflow.org, and notebooks.kubeflow.org. The adapters reference has the details.

Single sign-on

Register an OpenID Connect client for the console with the redirect URI https://platform.example.internal/oauth2/callback. Then store the client id, the client secret, and a random cookie secret for oauth2-proxy, which reads each of its flags from an environment variable named OAUTH2_PROXY_ followed by the flag's name in capitals, with underscores for hyphens:

$ kubectl -n fendlemark create secret generic oauth2-proxy \
    --from-literal=OAUTH2_PROXY_CLIENT_ID=fendlemark-console \
    --from-literal=OAUTH2_PROXY_CLIENT_SECRET="$OIDC_CLIENT_SECRET" \
    --from-literal=OAUTH2_PROXY_COOKIE_SECRET="$(openssl rand -base64 32 | tr -- '+/' '-_')"

3. Write the configuration

The app reads one file. Keep it in a ConfigMap; the secrets stay in the environment, under names the adapters already read.

apiVersion: v1
kind: ConfigMap
metadata:
  name: fendlemark-config
  namespace: fendlemark
data:
  fendlemark.yaml: |
    server:
      base_url: "https://platform.example.internal"
      data_dir: "/var/lib/fendlemark"

    console:
      title: "ML Platform"
      default_view: runs

    adapters:
      mlflow:
        enabled: true
        tracking_uri: "http://mlflow.mlflow.svc:5000"
      metaflow:
        enabled: true
        metadata_url: "http://metaflow-metadata.metaflow.svc:8080"
        ui_url: "https://metaflow-ui.example.internal"
      argo:
        enabled: true
        server: "https://argo-server.argo.svc:2746"
        namespaces: ["ml-training", "ml-batch", "team-ml", "team-risk"]
      kubeflow:
        enabled: true
        pipelines_url: "http://ml-pipeline.kubeflow.svc:8888"
        profiles: ["team-ml", "team-risk"]
        dashboard_url: "https://kubeflow.example.internal"
        katib: false
        notebooks: false

Three things are deliberate:

Unknown keys are errors, not warnings: a misspelt key under server, console, or lattice makes serve exit with code 1 at start-up instead of being ignored. Inside an adapter's block, a missing required option or a value of the wrong type stops it the same way, but a misspelt option name is not caught: katlb: false leaves Katib on. Check the effective configuration that fendlemark doctor prints.

4. Deploy the app and the proxy

One Deployment runs both containers. The app takes its configuration from the ConfigMap and its credentials from the Secrets in step 2; the proxy takes the OpenID Connect client.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: fendlemark
  namespace: fendlemark
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: fendlemark
  template:
    metadata:
      labels:
        app.kubernetes.io/name: fendlemark
    spec:
      serviceAccountName: fendlemark
      automountServiceAccountToken: false     # every credential arrives through the environment
      securityContext:
        runAsNonRoot: true
        runAsUser: 65532
        runAsGroup: 65532
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: fendlemark
          image: ghcr.io/tophergopher/fendlemark:0.9.12
          args: ["serve", "--config", "/etc/fendlemark/fendlemark.yaml"]
          envFrom:
            - secretRef:
                name: fendlemark-mlflow
            - secretRef:
                name: fendlemark-kubeflow
          env:
            - name: ARGO_TOKEN
              valueFrom:
                secretKeyRef:
                  name: fendlemark-argo
                  key: token
          readinessProbe:                     # through the proxy, which passes this path on
            httpGet:
              path: /readyz
              port: 4180
            periodSeconds: 10
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          volumeMounts:
            - name: config
              mountPath: /etc/fendlemark
              readOnly: true
            - name: data
              mountPath: /var/lib/fendlemark

        - name: oauth2-proxy
          image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.5
          args:
            - --provider=oidc
            - --oidc-issuer-url=https://sso.example.internal/realms/platform
            - --redirect-url=https://platform.example.internal/oauth2/callback
            - --email-domain=example.com
            - --upstream=http://127.0.0.1:7411/
            - --http-address=0.0.0.0:4180
            - --reverse-proxy=true
            - --trusted-proxy-ip=10.244.0.0/16  # your cluster's pod CIDR; step 5 admits only the ingress controller
            - --skip-provider-button=true
            - "--skip-auth-route=GET=^/(healthz|readyz)$"
          envFrom:
            - secretRef:
                name: oauth2-proxy
          ports:
            - name: http
              containerPort: 4180
          livenessProbe:
            httpGet:
              path: /ping
              port: http
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]

      volumes:
        - name: config
          configMap:
            name: fendlemark-config
        - name: data
          emptyDir: {}

The parts that are not boilerplate:

5. Expose the proxy through an Ingress

The Service points at the proxy's port, never at 7411, and a NetworkPolicy keeps everything but the ingress controller away from it:

apiVersion: v1
kind: Service
metadata:
  name: fendlemark
  namespace: fendlemark
spec:
  selector:
    app.kubernetes.io/name: fendlemark
  ports:
    - name: http
      port: 80
      targetPort: http
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: fendlemark
  namespace: fendlemark
spec:
  ingressClassName: internal          # your ingress controller's class
  tls:
    - hosts: ["platform.example.internal"]
      secretName: platform-tls
  rules:
    - host: platform.example.internal
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: fendlemark
                port:
                  name: http
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: fendlemark-ingress
  namespace: fendlemark
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: fendlemark
  policyTypes: ["Ingress"]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: ingress   # your ingress controller's namespace
      ports:
        - port: 4180

With the NetworkPolicy, other pods in the cluster cannot reach the proxy except through the Ingress and its TLS. Use your own ingress controller's class and namespace; Kubernetes labels every namespace with kubernetes.io/metadata.name. A NetworkPolicy takes effect only if your cluster's network plugin enforces it.

6. Check it

Wait for the rollout, then check that each adapter connected and the first reconciliation finished:

$ kubectl -n fendlemark rollout status deployment/fendlemark
deployment "fendlemark" successfully rolled out
$ kubectl -n fendlemark logs deployment/fendlemark -c fendlemark | grep -E 'adapter|lattice|reconciliation'
time=2026-10-29T10:02:11.204Z level=INFO msg="fendlemark starting" version=0.9.12 adapters=argo,kubeflow,metaflow,mlflow
time=2026-10-29T10:02:11.204Z level=INFO msg="adapter connected" adapter=argo took=0s
time=2026-10-29T10:02:11.204Z level=INFO msg="adapter connected" adapter=kubeflow took=0s
time=2026-10-29T10:02:11.204Z level=INFO msg="adapter connected" adapter=metaflow took=0s
time=2026-10-29T10:02:11.204Z level=INFO msg="adapter connected" adapter=mlflow took=0s
time=2026-10-29T10:02:14.117Z level=INFO msg="lattice reconciled" entities=6112 facets=7986 generation=1 took=97ms
time=2026-10-29T10:02:14.117Z level=INFO msg="adapter watching" adapter=argo
time=2026-10-29T10:02:14.117Z level=INFO msg="adapter watching" adapter=kubeflow

"Connected" means only that an adapter read its configuration; it makes no request until the first reconciliation. An adapter that cannot list, for a missing permission, a rejected token, or a certificate it does not trust, is named in a level=WARN line, msg="initial reconciliation had errors", which the grep above also shows.

A rollout that never finishes is almost always a pod that never becomes ready. To see what the probe sees, forward the proxy's port and ask:

$ kubectl -n fendlemark port-forward deployment/fendlemark 4180:4180 &
$ curl -s -o /dev/null -w '%{http_code}\n' http://localhost:4180/readyz
502

A 502 means the app is not listening yet: either the initial reconciliation is still running, or the container has exited, and its logs will say which. A 200 means the pod is ready.

Then open https://platform.example.internal and sign in. The Adapters view shows each adapter's health, latency, facet counts, and effective configuration, from the same checks that fendlemark adapters check and GET /api/v1/adapters run.

Things that will happen eventually

The console's own sign-in

Fendlemark does not integrate with identity providers. Its own sign-in, Respiratory Identity, is off by default: it needs a webcam at every desk and an enrolment for every person, three breaths each. Behind oauth2-proxy, the console admits whoever your identity provider admits, keeps them signed in for as long as the proxy's cookie lasts (--cookie-expire, a week by default), and asks nothing of anyone's lungs. If you enable Respiratory Identity anyway, mount a PersistentVolumeClaim at server.data_dir, where the enrolments are kept.

Further reading