Kubeflow + Argo CD: Making MLOps Work Like Charm

The problem: ML in production is mostly plumbing

A data scientist gets a model to 0.94 F1 in a notebook. Then reality hits. That notebook has to become a repeatable training pipeline, the resulting artifact has to be versioned, a serving endpoint has to be stood up, and the whole thing has to be re-runnable when data drifts next month. Along the way you accumulate a pile of bash scripts, hand-run kubectl applys, and tribal knowledge that lives in one engineer's terminal history.

The failure modes are always the same:

• Non-reproducibility — "it worked when I ran it" — no record of what was deployed, when, or from which commit.
• Configuration drift — someone hotfixes the cluster live; the next deploy silently reverts it.
• Fragile handoffs — the gap between "notebook" and "production" is a manual, error-prone relay race.
• No easy rollback — a bad model or config is a scramble, not a one-line revert.

Two CNCF-ecosystem tools solve two halves of this. Kubeflow gives you the ML workloads. Argo CD gives you a disciplined, Git-driven way to deploy and keep them in sync. Used together, they turn the mess above into a system.

What each tool actually solves

Watch: operationalizing ML with Kubeflow + GitOps

Before the hands-on, this KubeCon + CloudNativeCon talk is the best 30-minute overview of the exact journey this post describes — going from ad-hoc bash scripts to Kubeflow pipelines delivered through GitOps, at real scale.

“From Bash Scripts to Kubeflow and GitOps: Our Journey to Operationalizing ML at Scale” — KubeCon + CloudNativeCon (CNCF).

Reference architecture

The pattern is a GitOps loop. You never kubectl apply to production by hand. You change YAML in Git; Argo CD notices and reconciles the cluster. Kubeflow and every ML app it hosts are just Applications in that loop.

          commit / PR
   You ─────────────────────▶  Git repo (source of truth)
                                     │  kubeflow/            (platform manifests)
                                     │  pipelines/           (your ML pipelines)
                                     │  serving/             (KServe InferenceServices)
                                     │  apps/                (app-of-apps root)
                                     ▼
                              ┌─────────────┐   watches & reconciles
                              │   Argo CD   │ ───────────────────────▶  Kubernetes cluster
                              └─────────────┘   (self-heal + drift)          │
                                     ▲                                       ▼
                                     │        Kubeflow: Pipelines · KServe · Katib · Notebooks
                                     └──────────────  status / health  ──────────────┘

Deploy it, step by step

The following gets you from an empty cluster to Kubeflow delivered by Argo CD via GitOps. You'll need a Kubernetes cluster (v1.29+ recommended for current Kubeflow), kubectl, and kustomize.

Argo CD goes in first — it will deploy everything else.

bashkubectl create namespace argocd
kubectl apply -n argocd \
  -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# Wait for it to come up
kubectl -n argocd rollout status deploy/argocd-server

Grab the initial admin password and log in (port-forward for local access):

bashkubectl -n argocd port-forward svc/argocd-server 8080:443 &
argocd admin initial-password -n argocd            # prints the password
argocd login localhost:8080 --username admin --insecure

Fork or vendor the official kubeflow/manifests repo into your own Git repo. Pin a release tag — never track main for a platform this large. Your repo layout might look like:

repo layoutmy-mlops-platform/
├── kubeflow/            # vendored kubeflow/manifests @ v1.9.x (the "example" overlay)
├── pipelines/           # your compiled KFP pipelines
├── serving/             # KServe InferenceServices
└── apps/                # Argo CD Application definitions (app-of-apps)

Now tell Argo CD to reconcile the cluster to your Kubeflow manifests:

kubeflow-app.yamlapiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: kubeflow
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/<you>/my-mlops-platform.git
    targetRevision: main
    path: kubeflow          # the kustomize overlay
  destination:
    server: https://kubernetes.default.svc
  syncPolicy:
    automated:
      prune: true           # delete what Git no longer declares
      selfHeal: true        # revert out-of-band changes
    syncOptions:
      - ServerSideApply=true # required — Kubeflow CRDs are large
      - CreateNamespace=true
    retry:
      limit: 10             # Kubeflow's CRDs+CRs need a few passes to settle
      backoff: { duration: 20s, factor: 2, maxDuration: 5m }

bashkubectl apply -f kubeflow-app.yaml
argocd app sync kubeflow
argocd app wait kubeflow --health

Rather than manage each piece by hand, create one root Application that points at your apps/ directory. Every child — Kubeflow, your pipelines, model serving, monitoring — becomes a versioned, self-healing Application. Add a new capability by adding a file and merging a PR.

apps/root.yamlapiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: mlops-root
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/<you>/my-mlops-platform.git
    targetRevision: main
    path: apps              # a directory full of Application manifests
  destination:
    server: https://kubernetes.default.svc
  syncPolicy:
    automated: { prune: true, selfHeal: true }

With the platform live, deploying a model is a pull request. Compile a Kubeflow pipeline, commit it, and declare a KServe InferenceService that Argo CD will roll out — with canary traffic splitting built in:

serving/fraud-model.yamlapiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: fraud-detector
  namespace: kubeflow-user
spec:
  predictor:
    canaryTrafficPercent: 20        # send 20% to the new model first
    model:
      modelFormat: { name: sklearn }
      storageUri: gs://my-models/fraud/v7/

Bump v7 to v8 in a PR, merge, and Argo CD does the canary rollout. Roll back with git revert. That's the whole point.

Making it run like charm

Getting it working is step one. These practices are what keep it boring — in the best way:

• Sync waves for ordering. Annotate resources with argocd.argoproj.io/sync-wave so CRDs, then controllers, then workloads apply in order.
• Separate environments as overlays. overlays/dev, overlays/staging, overlays/prod — each its own Application, each promoted by a PR.
• Never commit secrets. Use Sealed Secrets, External Secrets Operator, or SOPS. Git is public-by-accident-prone; encrypt what matters.
• Automate image bumps. Argo CD Image Updater can open PRs when a new model or component image is published — GitOps all the way down.
• Watch the pipelines, not just the deploys. A green Argo CD sync means the manifest applied — not that the training job succeeded or the model is healthy. Monitor pipeline runs and serving latency separately.
• Make rollback a reflex. Because Git is the source of truth, a bad model version, a broken CRD upgrade, or a runaway job are all one git revert away.

Keep watching

Hands-on follow-ups from the wider community:

• KubeFlow Pipelines Zero to Hero — a realtime MLOps project
• Kubeflow MLOps: from notebook development to production inference
• Argo CD Tutorial for Beginners — GitOps CD for Kubernetes
• CNCF Projects Tour: Mastering GitOps with Argo (Kostis Kapelonis)