kcore has been accepted into the NVIDIA Inception program for startups. For a small team building a hypervisor, it comes at the point where GPU work stops being a side feature. Two things shipped in 0.7.0 make that the right time: one NVIDIA card can now be shared by several VMs, and a Slurm cluster runs on the hosts themselves.
One card, several VMs
Until now a GPU went to one VM. That is fine for a workstation guest and wasteful for inference, where most guests need a few gigabytes of video memory, not a whole card. kcore now slices an NVIDIA card into vGPUs. You turn a node into a vGPU host once, and each VM asks for a profile:
kctl node vgpu enable node-193 --host-card 0000:61:00.0 \
--url http://mirror.lan/NVIDIA-Linux-x86_64-595.91.04-vgpu-kvm-aie.run \
--sha256 2bc7f9be... --file NVIDIA-Linux-x86_64-595.91.04-vgpu-kvm-aie.run
kctl create vm infer-01 --vgpu A2-4C ...
The node downloads the NVIDIA Virtual GPU Manager, checks its SHA256, and loads it from
its Nix configuration. The .run file never goes into git. If the card is
still on nouveau, the command schedules the reboot, and a reboot is refused
while a VM is meant to be running unless you force it. Leave the target node empty and
the controller places the VM on a ready node that can supply the profile.
The same record is kind: NodeVgpu for kctl apply,
kcore_vgpu_host and kcore_vm.vgpus in the Terraform provider,
and a node-vgpu kind in the MCP server, so an agent can see which slices
are free. The vGPU manager needs an NVIDIA AI Enterprise entitlement.
Intel and AMD cards use plain SR-IOV with no licence. intel-2vf is half an
Intel card and radeon-4vf a quarter of an AMD one, with a function count
per card. The manifest is the same as for NVIDIA.
The patches go to Cloud Hypervisor
kcore runs guests on Cloud Hypervisor.
NVIDIA's vGPU manager was written against QEMU, so making an A2 slice boot took eight
small patches to Cloud Hypervisor and its vfio-ioctls crate. Each one
fixes a place where Cloud Hypervisor and QEMU behave differently:
- Install guest RAM mappings before the VFIO device is opened, because the vGPU manager takes its snapshot of guest memory at that moment.
- Read region flags from the full-size
GET_REGION_INFOanswer, where the driver reports that BAR0 can be mapped. - Map the driver's sparse mmap areas even when the BAR also holds the MSI-X tables.
- Write BAR addresses through to the device, and read the identity registers from the device rather than a shadow copy.
- Warn instead of failing when
KVM_SET_TSC_KHZreturnsEINVAL. This one is not about GPUs: it lets a VM live-migrate from a host with a slower TSC to a faster one, as QEMU already does.
Carrying patches means rebuilding Cloud Hypervisor on every release, and it means
kcore users run code nobody upstream has reviewed. So the plan is to submit them. The
two vfio-ioctls changes go to rust-vmm first, because one Cloud Hypervisor
patch calls the new function. The rest follow, each opened as an issue first, as Cloud
Hypervisor's contribution guide asks. Each one will say plainly that it was tested on
one NVIDIA A2. The eighth patch keeps a QEMU-style -uuid on the process
command line because the vGPU manager reads it from there. That is a workaround for
the vendor manager, not something for Cloud Hypervisor's main, so kcore
keeps carrying it while we look for a cleaner fix.
Slurm on the bare metal
HPC and training jobs do not want a VM between them and the hardware. A
SlurmCluster runs Slurm as kcoreOS host services: a controller, an
optional backup controller, login nodes, accounting with slurmdbd and
MariaDB, and compute nodes.
kind: SlurmCluster
metadata:
name: hpc-gpu
spec:
clusterName: hpc-gpu
controller: { nodeId: node-101, address: 10.0.0.11, iface: enp6s0 }
accounting: { enabled: true, nodeId: node-104 }
gpuDriver: true
compute:
- nodeId: node-103
address: 10.0.0.13
iface: enp6s0
partitions: [batch, gpu]
gres: auto
kctl slurm plan -f hpc-gpu.yaml
kctl apply -f hpc-gpu.yaml
Compute nodes are cordoned, so the VM scheduler leaves them to slurmd.
Plan refuses a compute node that still hosts guests. The controller, login, and
accounting roles can share a node with VMs. gres: auto writes
gres.conf from the GPU inventory the controller already has, and plan
fails rather than writing a config that cannot start if the NVIDIA driver is off.
Cluster updates know Slurm's version rules: accounting first, then the controller, then compute, then login, and each compute node is drained before its rebuild. The munge key and the database password are generated on the controller and never returned by the API. The dashboard, the MCP server, and the Terraform provider cover the same resource, so one kcore estate can hold VMs, vGPU guests, and a batch cluster.
If you run inference or HPC workloads on VMware or bare Slurm today and want to try this on your own hardware, write to team@tacconiconsulting.com. The GPU guide covers passthrough and vGPU, and the Slurm guide has the full manifest reference.