Requirements

Before you deploy CosmicAC on your host machine, your Kubernetes cluster must meet the requirements on this page so CosmicAC can run jobs on it.

The CosmicAC stack includes a GPU worker, cosmicac-wrk-server-k8s-nvidia. The worker is a control-plane agent that connects to your cluster's Kubernetes API. It creates GPU virtual machines as KubeVirt VMIs, along with their supporting resources. The worker doesn't install the operators, drivers, or node configuration it relies on. You install and maintain them on your cluster. For how the worker connects to your cluster and runs jobs, see Deployment architecture.

Use the readiness checklist that follows to confirm your cluster is ready, not to build one. Any version or value on this page comes from the CosmicAC reference cluster. Treat it as a reference point, not a hard requirement, unless the page marks it as required.

Readiness checklist

Your cluster is ready when all the following are true:

• Kubernetes version supported by KubeVirt 1.7 and CDI 1.62, with v1.33.x as the reference
• At least one GPU node ready, with whole-device VFIO passthrough configured
• NVIDIA driver, Fabric Manager, and NVSwitch healthy on each GPU node
• GPU resource advertised on each node as nvidia.com/<gpuType>, plus mellanox.com/<ib> for multi-node
• KubeVirt installed and healthy, with the HostDevices feature gate enabled
• CDI installed and healthy, with a scratch StorageClass set
• KubeVirt GPU device plugin daemonset running and advertising the GPU resource name
• KubeVirt CR permittedHostDevices listing your GPU and IB PCI IDs
• A working StorageClass named local-path available, ideally with WaitForFirstConsumer
• Secret ghcr-pull-cred with its five keys in the workload namespace
• ConfigMap bootstrap-cm in the workload namespace, for in-guest container bootstrap
• Worker kubeconfig or ServiceAccount granted the RBAC in Section 7
• Multus, Whereabouts, OVS CNI, and subcluster-manager-sa installed

1. Cluster baseline

2. GPU node requirements

You need at least one GPU node. The CosmicAC reference cluster uses eight NVIDIA H100 80 GB SXM GPUs and eight ConnectX-7 InfiniBand NICs per node, but a single GPU is the minimum for a single-node workload.

2.1 Node labels

Each GPU node must carry the following labels:

The reference cluster also carries a node-role.kubernetes.io/gpu label, but the worker's VMI spec doesn't select it. The label is informational and not required for the worker.

2.2 Whole-device passthrough (VFIO)

VMIs claim whole GPUs and IB PFs through PCI passthrough, so each GPU node must bind those devices to vfio-pci at the host level. Each GPU node must meet the following:

• IOMMU enabled and the host GPU driver kept off the passed-through devices
• Every GPU PCI ID and every IB PF PCI ID bound to vfio-pci
• NVSwitch left on its native driver, with Fabric Manager healthy so NVLink works across the passed-through GPUs

The NVIDIA Network Operator isn't required for the current VFIO and IB mechanism. IB PFs appear as plain passthrough PCI devices, not through the Network Operator.

2.3 Verify resource advertisement

After host preparation and the KubeVirt GPU device plugin from Section 3, each GPU node must advertise the GPU as an allocatable resource named nvidia.com/<gpuType>. For multi-node jobs, it must also advertise the IB PF as mellanox.com/<ibName>.

On the CosmicAC reference cluster, each GPU node advertises the following resources:

nvidia.com/GH100_H100_SXM5_80GB = 8
mellanox.com/cx7_ib_pf          = 8
nvidia.com/gpu                  = 0   # expected: the worker requests the passthrough name, not nvidia.com/gpu

nvidia.com/gpu can legitimately be 0. The worker requests nvidia.com/<gpu.type> and mellanox.com/<ib_name>, not the generic nvidia.com/gpu, so the advertised name must match what the worker expects.

3. GPU and virtualization platform

This section lists the hard requirements for the worker's passthrough path, then the reference implementation that satisfies them.

3.1 Hard requirements

3.2 Reference implementation

On the CosmicAC reference cluster, the following components meet these requirements:

The GPU Operator installs the nvidia and nvidia-cdi RuntimeClasses on the reference cluster, but they aren't a prerequisite. The worker's VMIs don't set runtimeClassName.

3.3 Required KubeVirt permittedHostDevices

KubeVirt allows passthrough only for PCI devices you explicitly permit. The kubevirt/kubevirt CR must permit your GPU and IB PFs in its permittedHostDevices. The reference value is as follows:

spec:
  configuration:
    permittedHostDevices:
      pciHostDevices:
        - pciVendorSelector: "10DE:2330"          # NVIDIA H100 SXM5 80GB
          resourceName: nvidia.com/GH100_H100_SXM5_80GB
          externalResourceProvider: true          # advertised by the KubeVirt GPU device plugin
        - pciVendorSelector: "15B3:1021"          # Mellanox ConnectX-7 IB PF
          resourceName: mellanox.com/cx7_ib_pf
          externalResourceProvider: false

On different hardware, replace the pciVendorSelector PCI vendor:device pair and the resourceName to match your GPUs and NICs. The resourceName must match what the worker requests. See Section 2.3.

4. Storage

The worker creates DataVolumes for VM root disks and optional PersistentVolumeClaims for external data disks, with storageClassName: local-path. Your cluster must meet the following requirements:

• A working StorageClass named local-path must exist. It doesn't have to be the cluster default, but prefer WaitForFirstConsumer so the scheduler places the VMI on a GPU node before it provisions the disk locally.
• CDI must have a scratch StorageClass set for image imports. Reference value scratchSpaceStorageClass: local-path with the HonorWaitForFirstConsumer feature gate enabled.

On the CosmicAC reference cluster, each node uses the local-path provisioner rancher.io/local-path with WaitForFirstConsumer. Provide enough local NVMe storage. VM root disks are commonly 150Gi or larger, so size the storage for your model cache.

5. Registry credentials in ghcr-pull-cred

The worker pulls private images from the GitHub Container Registry (GHCR) at two stages. Each stage reads different keys from the same secret in your workload namespace:

KubeVirt references the secret as the vmImagePullSecret, named ghcr-pull-cred by default. The credentials are a GitHub personal access token (PAT) with at least read:packages, SSO-authorized for the tetherto org. The secret holds all five keys:

kubectl -n <workload-namespace> create secret generic ghcr-pull-cred \
  --from-literal=accessKeyId='<github-username>' \
  --from-literal=secretKey='<github-PAT>' \
  --from-literal=DOCKER_REGISTRY='ghcr.io' \
  --from-literal=DOCKER_USERNAME='<github-username>' \
  --from-literal=DOCKER_TOKEN='<github-PAT>'

The resulting secret holds these five keys:

5.1 Bootstrap ConfigMap

Workloads that boot the in-guest inference container mount a bootstrap-cm ConfigMap as a CD-ROM to install the in-guest docker bootstrap unit. It must exist in the workload namespace before the worker starts. The ConfigMap holds two keys. docker-bootstrap.sh installs Docker, optionally prepares an external PVC disk, logs into the registry, and pulls and runs the container. docker-bootstrap.service is the systemd unit that runs the script on first boot.

The worker repository ships the canonical manifest at setup/cfg-create.yml. Its namespace must match your workload namespace.

kubectl apply -f setup/cfg-create.yml

The following is the complete ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: bootstrap-cm
  namespace: <workload-namespace>   # e.g. cosmic-ac

data:
  docker-bootstrap.sh: |
    #!/bin/bash
    set -eu

    # Helper: wait for block device
    wait_for_dev() { for i in {1..60}; do [ -b "$1" ] && return 0; sleep 1; done; echo "ERROR: $1 not found"; lsblk -b; exit 1; }

    apt-get update && apt-get install -y curl ca-certificates docker.io parted e2fsprogs util-linux
    systemctl enable --now docker

    # Setup PVC disk
    ROOT_DISK="$(findmnt -n -o SOURCE / | sed 's/[0-9]*$//' | xargs basename)"
    [ -f /etc/default/docker-bootstrap ] && source /etc/default/docker-bootstrap
    MNT="${MNT:-}"

    if [ -n "${MNT}" ]; then
      PVC_DISK_NAME="$(lsblk -b -dn -o NAME,TYPE,SIZE | awk -v r="${ROOT_DISK}" '$2=="disk" && $1!=r && $1!~/^(sr|loop)/ && $3>=107374182400 {print $1,$3}' | sort -k2,2nr | head -n1 | awk '{print $1}')"
      [ -z "${PVC_DISK_NAME}" ] && { echo "ERROR: Could not find PVC disk (>=100GiB)"; lsblk -b; exit 1; }

      DISK="/dev/${PVC_DISK_NAME}"
      PART="${DISK}1"
      wait_for_dev "${DISK}"

      [ ! -b "${PART}" ] && { parted -s "${DISK}" mklabel gpt mkpart primary ext4 1MiB 100%; udevadm settle || true; }
      wait_for_dev "${PART}"
      blkid "${PART}" | grep -q 'TYPE=' || mkfs.ext4 -F "${PART}"

      mkdir -p "${MNT}"
      UUID="$(blkid -s UUID -o value "${PART}")"
      grep -q "${UUID}" /etc/fstab || echo "UUID=${UUID} ${MNT} ext4 defaults,nofail 0 2" >> /etc/fstab
      mount -a
    else
      echo "MNT not set, skipping external disk setup"
    fi

    # Docker login and pull
    SECRET_MNT="/mnt/registrycred"
    mkdir -p "${SECRET_MNT}"
    wait_for_dev /dev/sr0
    mount -t iso9660 -o ro /dev/sr0 "${SECRET_MNT}"

    REGISTRY="$(cat "${SECRET_MNT}/DOCKER_REGISTRY")"
    USERNAME="$(cat "${SECRET_MNT}/DOCKER_USERNAME")"
    DOCKER_CONFIG="/run/docker-config"
    mkdir -p "${DOCKER_CONFIG}" && chmod 700 "${DOCKER_CONFIG}"

    cat "${SECRET_MNT}/DOCKER_TOKEN" | docker login "${REGISTRY}" -u "${USERNAME}" --password-stdin
    IMAGE_FULL="${IMG:-${USERNAME}/cosmicac-wrk-agent-inference:latest}"
    docker pull "${IMAGE_FULL}"

    docker logout "${REGISTRY}" || true
    rm -rf "${DOCKER_CONFIG}" && umount "${SECRET_MNT}" || true

    # Run container if not already running
    CONTAINER_NAME="${NAME:-cosmicac-container}"
    if docker ps --format '{{.Names}}' | grep -q "^${CONTAINER_NAME}$"; then
      echo "Container ${CONTAINER_NAME} already running"
    elif docker ps -a --format '{{.Names}}' | grep -q "^${CONTAINER_NAME}$"; then
      docker start "${CONTAINER_NAME}"
    elif [ -n "${IMG}" ]; then
      BASE_CMD="docker run -d --name ${CONTAINER_NAME} --init --restart unless-stopped --gpus all --ipc=host"
      [ -n "${MNT}" ] && BASE_CMD="${BASE_CMD} -v ${MNT}:${MNT}"

      if [ "${JOB_TYPE:-}" = "GPU_CONTAINER" ]; then
        BASE_CMD="${BASE_CMD} --cap-add=NET_ADMIN --device /dev/net/tun"
      fi

      if [ -n "${ENVS}" ]; then
        IFS=';' read -ra ENV_ARRAY <<< "${ENVS}"
        for env_pair in "${ENV_ARRAY[@]}"; do
          [[ "$env_pair" != *=* ]] && continue
          ENV_KEY="${env_pair%%=*}"
          ENV_VAL="${env_pair#*=}"
          if [[ "$ENV_VAL" =~ ^[\{\[] ]]; then
            # Convert JS-style object to JSON (single sed pass)
            JSON_VAL=$(sed -E '
              s/([{,])([a-zA-Z0-9_]+):/\1"\2":/g;
              s/:([0-9]+)/: \1/g
            ' <<< "$ENV_VAL")
            BASE_CMD+=" -e ${ENV_KEY}='${JSON_VAL}'"
          else
            BASE_CMD+=" -e ${ENV_KEY}=${ENV_VAL}"
          fi
        done
      fi
      eval "${BASE_CMD} ${IMG}"
    fi

    echo "Bootstrap completed successfully."
  docker-bootstrap.service: |
    [Unit]
    Description=Deferred Docker Bootstrap (login + pull + run)
    Wants=network-online.target
    After=network-online.target multi-user.target

    [Service]
    Type=oneshot
    ExecStart=/usr/local/bin/docker-bootstrap.sh
    RemainAfterExit=yes

    [Install]
    WantedBy=multi-user.target

Verify it exists:

kubectl get configmap bootstrap-cm -n <workload-namespace>   # must exist

6. Required container images

Your cluster and the VM guests must be able to pull the following:

The ghcr.io/tetherto/* images are private. Access them with the ghcr-pull-cred PAT from Section 5.

7. Worker access and RBAC

The worker authenticates to your API server with a kubeconfig. The identity behind that kubeconfig needs two sets of permissions.

Namespace-scoped permissions cover the resources the worker creates and manages in the workload namespace, and, for overlay networking, in kube-system:

Cluster-scoped permissions let the worker report cluster capacity through getClusterResources, which getInfo and the getClusterResources RPC both use. That call lists all nodes and all pods cluster-wide to compute GPU, CPU, and memory totals and usage:

The following notes apply:

• pods needs namespaced list and get so the worker can poll VMI launcher pods for status, and cluster-wide list for getClusterResources.
• nodes is cluster-scoped, so it requires a ClusterRole.
• For overlay networking, configmaps also needs update and patch. The worker rewrites the WireGuard peers ConfigMap as clients join or leave.

The namespaced set binds through a Role and RoleBinding in the workload namespace. A ClusterRole and ClusterRoleBinding cover the cluster-scoped node and pod reads along with the kube-system objects that overlay networking uses. The grant follows least privilege, and the worker never needs cluster-admin.

8. Optional overlay networking add-ons

You need these add-ons only for workloads that use per-instance overlay networking, an isolated OVS/VXLAN overlay reachable through a WireGuard gateway. Only some workloads use it today, and that use keeps growing. Treat it as an optional requirement tier rather than something tied to one workload type. Skip this section if you don't need per-instance network isolation.

Your cluster needs the following outcomes:

These add-ons layer on top of your primary CNI. They don't replace it.

9. Observability and logs

CosmicAC doesn't include an observability stack. It doesn't deploy or manage Prometheus, Loki, Grafana, or any metrics or log storage, and it doesn't need them to run jobs.

The Model Health page and the CLI show live metrics from the running service. These metrics reset when the service restarts, and CosmicAC doesn't keep them. CosmicAC doesn't store historical metrics or collect logs.

To collect historical metrics or logs, run an observability stack yourself. Prometheus and Loki are common choices for metrics and logs. You install, operate, and maintain the stack. CosmicAC doesn't provide or configure it for you.

10. Verification checklist

Run these checks from a workstation with kubectl and virtctl pointed at the cluster.

9.1 Base platform

# Nodes Ready + control plane healthy; k8s version supported by KubeVirt/CDI
kubectl get nodes -o wide
kubectl version -o json | jq -r '.serverVersion.gitVersion'

# Per GPU node: GPU (and IB) advertised under the name the worker requests
kubectl get nodes -o json | jq -r '
  .items[] | .metadata.name + "  gpu=" +
  (.status.allocatable["nvidia.com/GH100_H100_SXM5_80GB"] // "0") + "  ib=" +
  (.status.allocatable["mellanox.com/cx7_ib_pf"] // "0")'

# KubeVirt + CDI healthy; HostDevices feature gate on
kubectl -n kubevirt get kubevirt kubevirt -o jsonpath='{.status.observedKubeVirtVersion}{"\n"}'
kubectl -n kubevirt get kubevirt kubevirt -o jsonpath='{.spec.configuration.developerConfiguration.featureGates}{"\n"}'  # includes HostDevices
kubectl -n cdi get cdi cdi -o jsonpath='{.status.observedVersion}{"\n"}'

# KubeVirt GPU device plugin advertising the passthrough resource
kubectl -n kube-system get ds nvidia-kubevirt-gpu-dp-daemonset

# permittedHostDevices includes your GPU/IB
kubectl -n kubevirt get kubevirt kubevirt -o jsonpath='{.spec.configuration.permittedHostDevices}{"\n"}' | jq .

# A working local-path StorageClass exists (need not be the default)
kubectl get sc

# Workload-namespace prerequisites
kubectl -n <workload-namespace> get secret ghcr-pull-cred
kubectl -n <workload-namespace> get configmap bootstrap-cm

# Device-admission smoke test: a VMI that CLAIMS a full GPU is admitted + scheduled.
# NOTE: with no boot disk this only proves device admission + scheduling, NOT a
# working guest. For a real boot test, add a DataVolume root disk.
cat <<'EOF' | kubectl apply -f -
apiVersion: kubevirt.io/v1
kind: VirtualMachineInstance
metadata:
  name: gpu-smoke
  namespace: default
spec:
  domain:
    devices:
      gpus:
        - deviceName: nvidia.com/GH100_H100_SXM5_80GB
          name: gpu0
    resources:
      requests: { memory: 8Gi }
  volumes: []
EOF
kubectl get vmi gpu-smoke -w     # reaches Scheduled/Running (device admitted), then:
kubectl delete vmi gpu-smoke

The checks pass when nodes are Ready, each GPU node reports the expected gpu= and ib= counts, KubeVirt is healthy with HostDevices on, CDI is healthy, a working local-path StorageClass exists, ghcr-pull-cred and bootstrap-cm are present, and the cluster admits and schedules the smoke VMI.

9.2 Optional overlay networking

# Multus / Whereabouts / OVS daemonsets Ready in kube-system
kubectl -n kube-system get ds | grep -iE 'multus|whereabouts|ovs'

# NAD CRD present
kubectl get crd network-attachment-definitions.k8s.cni.cncf.io

# Subcluster manager SA + RBAC present
kubectl -n kube-system get sa subcluster-manager-sa
kubectl get clusterrole subcluster-manager-role
kubectl get clusterrolebinding subcluster-manager-binding

# LoadBalancer can allocate an external IP for UDP services (provider-specific)