GPU passthrough

A VM takes a named GPU from the node that will run it. You ask for radeon0 or nvidia1, not a PCI address. The node binds that card and its audio function to the guest.

This is whole-device passthrough. The guest gets the physical card, so the GPU cannot move live the way CPU and memory can. Moving the VM stops it, attaches a GPU on the destination, and boots the guest there. The disk still has to be shared Ceph RBD. See VM migration.

See the GPUs

On the machine you are logged into, read the local PCI bus. Across the cluster, read the inventory each node agent publishes.

kctl gpu list --local
kctl gpu list
kctl gpu list --node node-a

Names are per node. radeon0 is that node's lowest-address AMD GPU, radeon1 the next. NVIDIA cards are nvidia0, Intel graphics are intel0. A row is free, in-use, or blocked.

State	Meaning
`free`	The card can be assigned. Its IOMMU group is only the GPU and its audio function.
`in-use`	A VM already owns this name on this node.
`blocked`	The IOMMU group also contains USB, SATA, a bridge, or another device the host cannot give away. An integrated GPU that shares a group with the host's disks is blocked.

Assign a GPU

--target-node is required. --gpu radeon with no number takes the first free Radeon on that node. Repeat --gpu to attach more than one card. A name belongs to one VM.

kctl create vm gpu-guest \
  --target-node node-a \
  --gpu radeon0 \
  --cpu 4 --memory 8G \
  --image https://example.com/debian.raw \
  --image-sha256 <sha256> \
  --storage-backend ceph \
  --storage-size-bytes 21474836480

apiVersion: kcore/v1
kind: VM
metadata:
  name: gpu-guest
spec:
  cpu: 4
  memoryBytes: 8589934592
  targetNode: node-a
  gpus:
    - radeon0
  storageBackend: ceph
  storageSizeBytes: 21474836480

--pci 0000:03:00.0 still works when you need a raw address. Prefer --gpu.

Move the VM

Leave --gpu off to take a free GPU of the same family on the destination, one for each GPU the VM has now. Name a card when you want a specific one. Node drain uses the same automatic pick. If nothing compatible is free, the VM stays where it is.

kctl migrate vm gpu-guest --target-node node-b
kctl migrate vm gpu-guest --target-node node-b --gpu radeon1

The guest disk is not copied. Cold migration of a GPU VM needs the same shared RBD disk as any other cross-node move. See kcore SAN (Ceph).

Limits

The GPU itself is not live-migrated. CPU and memory of a GPU VM are not streamed to the new host while the card stays attached.
A blocked GPU cannot be assigned. Fix is a discrete card whose IOMMU group is only that card.
Changing the GPU list on a later apply is rejected. Migrate, or delete the VM and create it again.
The host loses the card while the guest has it. The device stays on vfio-pci after the guest stops until you bind the host driver again.

What the node does at start (VFIO bind, kernel parameters): docs/gpu-passthrough.md.