coder/docs/admin/integrations/multiple-kube-clusters.md

Additional clusters

With Coder, you can deploy workspaces in additional Kubernetes clusters using different authentication methods in the Terraform provider.

Option 1) Kubernetes contexts and kubeconfig

First, create a kubeconfig file with multiple contexts.

kubectl config get-contexts

CURRENT   NAME                        CLUSTER
          workspaces-europe-west2-c   workspaces-europe-west2-c
*         workspaces-us-central1-a    workspaces-us-central1-a

Kubernetes control plane

If you deployed Coder on Kubernetes, you can attach a kubeconfig as a secret.

This assumes Coder is deployed on the coder namespace and your kubeconfig file is in ~/.kube/config.

kubectl create secret generic kubeconfig-secret -n coder --from-file=~/.kube/config

Modify your helm values to mount the secret:

coder:
  # ...
  volumes:
    - name: "kubeconfig-mount"
      secret:
        secretName: "kubeconfig-secret"
  volumeMounts:
    - name: "kubeconfig-mount"
      mountPath: "/mnt/secrets/kube"
      readOnly: true

Upgrade Coder with these new values.

VM control plane

If you deployed Coder on a VM, copy the kubeconfig file to /home/coder/.kube/config.

Create a Coder template

You can start from our example template. From there, add template parameters to allow developers to pick their desired cluster.

# main.tf

data "coder_parameter" "kube_context" {
  name         = "kube_context"
  display_name = "Cluster"
  default      = "workspaces-us-central1-a"
  mutable      = false
  option {
    name  = "US Central"
    icon  = "/emojis/1f33d.png"
    value = "workspaces-us-central1-a"
  }
  option {
    name  = "Europe West"
    icon  = "/emojis/1f482.png"
    value = "workspaces-europe-west2-c"
  }
}

provider "kubernetes" {
  config_path    = "~/.kube/config" # or /mnt/secrets/kube/config for Kubernetes
  config_context = data.coder_parameter.kube_context.value
}

Option 2) Kubernetes ServiceAccounts

Alternatively, you can authenticate with remote clusters with ServiceAccount tokens. Coder can store these secrets on your behalf with managed Terraform variables.

Alternatively, these could also be fetched from Kubernetes secrets or even Hashicorp Vault.

This guide assumes you have a coder-workspaces namespace on your remote cluster. Change the namespace accordingly.

Create a ServiceAccount

Run this command against your remote cluster to create a ServiceAccount, Role, RoleBinding, and token:

kubectl apply -n coder-workspaces -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: coder-v2
---
apiVersion: v1
kind: Secret
metadata:
  name: coder-v2
  annotations:
    kubernetes.io/service-account.name: coder-v2
type: kubernetes.io/service-account-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: coder-v2
rules:
  - apiGroups: ["", "apps", "networking.k8s.io"]
    resources: ["persistentvolumeclaims", "pods", "deployments", "services", "secrets", "pods/exec","pods/log", "events", "networkpolicies", "serviceaccounts"]
    verbs: ["create", "get", "list", "watch", "update", "patch", "delete", "deletecollection"]
  - apiGroups: ["metrics.k8s.io", "storage.k8s.io"]
    resources: ["pods", "storageclasses"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: coder-v2
subjects:
  - kind: ServiceAccount
    name: coder-v2
roleRef:
  kind: Role
  name: coder-v2
  apiGroup: rbac.authorization.k8s.io
EOF

The output should be similar to:

serviceaccount/coder-v2 created
secret/coder-v2 created
role.rbac.authorization.k8s.io/coder-v2 created
rolebinding.rbac.authorization.k8s.io/coder-v2 created

2. Modify the Kubernetes template

You can start from our example template.

variable "host" {
  description = "Cluster host address"
  sensitive   = true
}

variable "cluster_ca_certificate" {
  description = "Cluster CA certificate (base64 encoded)"
  sensitive   = true
}

variable "token" {
  description = "Cluster CA token (base64 encoded)"
  sensitive   = true
}

variable "namespace" {
  description = "Namespace"
}

provider "kubernetes" {
  host                   = var.host
  cluster_ca_certificate = base64decode(var.cluster_ca_certificate)
  token                  = base64decode(var.token)
}

Create Coder template with managed variables

Fetch the values from the secret and pass them to Coder. This should work on macOS and Linux.

To get the cluster address:

kubectl cluster-info
Kubernetes control plane is running at https://example.domain:6443

export CLUSTER_ADDRESS=https://example.domain:6443

To fetch the CA certificate and token:

export CLUSTER_CA_CERTIFICATE=$(kubectl get secrets coder-v2 -n coder-workspaces -o jsonpath="{.data.ca\.crt}")

export CLUSTER_SERVICEACCOUNT_TOKEN=$(kubectl get secrets coder-v2 -n coder-workspaces -o jsonpath="{.data.token}")

Create the template with these values:

coder templates push \
    --variable host=$CLUSTER_ADDRESS \
    --variable cluster_ca_certificate=$CLUSTER_CA_CERTIFICATE \
    --variable token=$CLUSTER_SERVICEACCOUNT_TOKEN \
    --variable namespace=coder-workspaces

If you're on a Windows machine (or if one of the commands fail), try grabbing the values manually:

# Get cluster API address
kubectl cluster-info

# Get cluster CA and token (base64 encoded)
kubectl get secrets coder-service-account-token -n coder-workspaces -o jsonpath="{.data}"

coder templates push \
    --variable host=API_ADDRESS \
    --variable cluster_ca_certificate=CLUSTER_CA_CERTIFICATE \
    --variable token=CLUSTER_SERVICEACCOUNT_TOKEN \
    --variable namespace=coder-workspaces