High availability for VMs
When a host stops heartbeating, the controller fences its storage, then restarts its protected VMs on the surviving hosts. Nobody has to do anything. This page covers what is protected, how fast it happens, the knobs, and the cases HA does not cover.
What is protected
| VM storage | When the host dies |
|---|---|
ceph (kcore SAN) | Protected. The image is fenced, then the VM restarts on another host. |
zfs with a replica node | Protected, with a stated data-loss window. The VM starts on the replica node from the last common snapshot. |
zfs without a replica, lvm, filesystem | Not protected. The disk is on the dead host, so the VM stays there until the host comes back or you restore from backup. |
The protection level shows on every VM listing: kctl get vms, the dashboard, and the API (ha_protection). You do not find out after the outage.
What happens when a host dies
- Detect. Node agents heartbeat every 15 seconds. A host with no heartbeat for
nodeTimeoutSecs(default 30) is marked lost. A detector checks everydetectIntervalSecs(default 5). - Fence. The controller asks a healthy Ceph member to blocklist the lost host’s clients, per image and by address range. A host that returns cannot write to an image that has moved. Nothing is reassigned until the fence holds.
- Choose what restarts. The controller restarts what was running at the last sync, plus anything with
autoStart. A VM that was stopped stays stopped. - Order. VMs restart by
restartPriority(high,medium,low), then by memory, largest first. Priority decides who gets capacity first when capacity is short. - Start. Each target host gets its configuration once and starts the guests. The event trail records detect, fence, reassign, and apply, with durations.
In a lab run with Ceph at the older 90-second timeout, the guest was booted on its new host about 20 seconds after the fence. With the 30-second default, expect the fence about 30 to 35 seconds after the last heartbeat, plus guest boot time. Measure your own cluster: kctl get ha-events carries failover_duration_seconds for each move.
A returning host cannot start a stale guest
A host that rebooted boots its old NixOS generation, where the moved VM is still an autoStart unit. kcore closes that path two ways.
- Placement gate. Every
kcore-vm-*unit waits forkcore-placement-confirmed.target. The node agent reaches it only after a controller confirms which VMs belong on this host. A VM the controller no longer assigns here is stopped and masked. A Ceph VM stays held if no controller answers. Non-Ceph VMs start afterha.placementTimeoutSecs(default 300) in the node agent configuration, because nobody else can run them. - Fence by address. The blocklist covers the host’s whole address range, so a client whose watch already expired is still refused.
Isolation response
A host that lost every controller but still holds running guests would keep them with a fenced disk. After ha.isolationTimeoutSecs (default 60, node agent configuration), the node agent powers off protected Ceph guests it can no longer write, but only when Ceph shows the host is fenced or Ceph itself is unreachable. Local-disk VMs are left alone. Set isolationResponse to leaveRunning (controller failover block, and ha block on the node) if your controller link is flakier than your storage network. The controller-side fence protects the image either way.
Admission control
kcore refuses a placement that would leave protected VMs without a surviving host. hostFailuresToTolerate defaults to 1. For the largest hosts, the remaining hosts must have free memory and CPU for their protected VMs, counting overcommit. CreateVm, SetVmDesiredState, MigrateVm, and DrainNode run the check and answer FAILED_PRECONDITION naming the shortfall. A cluster too small to reserve that capacity is exempt. --ignore-admission-control overrides the check, and the override is audited.
kctl get ha # failover on or off, lease holder, tolerated host failures, headroom
kctl get ha-events # detected, fenced, reassigned, applied, failed, skipped
kctl get ha-events --node node-3
kctl get cluster-health
Redundant failover
With replicated controllers, one controller holds a failover lease and runs the detector and the mover. The lease is a replicated register, renewed every leaseRenewSecs (default 5). If the holder dies, another controller claims it after expiry and resumes from the last recorded event instead of fencing twice. A controller whose replication has fallen behind steps down.
Configure it
Controller settings live in controller.yaml. These are the defaults.
failover:
enabled: true
nodeTimeoutSecs: 30 # at least two heartbeat intervals (30)
detectIntervalSecs: 5
isolationResponse: powerOff # or leaveRunning
leaseRenewSecs: 5
admissionControl:
hostFailuresToTolerate: 1 # 0 to 5; 0 turns it off
Per VM, in a manifest or on kctl create vm:
kind: VM
metadata:
name: db-1
spec:
cpu: 8
memoryBytes: "16G"
storageBackend: ceph
storageSizeBytes: "200G"
restartPriority: high # --restart-priority
antiAffinity: db-tier # --anti-affinity; spreads the group over hosts
vmMonitoring: restart # --vm-monitoring off|restart; restart when the guest stops answering
vmMonitoringSecs: 120 # --vm-monitoring-secs, at least 30
A ZFS VM can keep an asynchronous replica on another node. The listing shows it as an asynchronous replica instead of unprotected.
kctl create vm app-1 --storage-backend zfs \
--replica-node node-2 --replica-interval 300 ... # 60 s to 1 day
Webhooks fire for node.lost, vm.failover.started, vm.failover.completed, and vm.failover.failed, and vm.migrated carries mode: failover.
What HA does not do
- It does not protect
lvm,filesystem, or unreplicatedzfsVMs. That is the same rule as shared-datastore HA elsewhere. The listing says so. - A ZFS replica is asynchronous. You lose writes since the last send. The interval is the recovery point.
- It is not full vSphere HA parity. There is no proactive HA on hardware health and no per-VM network isolation policy.
- Passthrough GPUs and PCI devices do not fail over by themselves. The destination needs a free matching device, and the VM must be remapped.
- Heartbeat detection is a heuristic. A host that is slow but alive for longer than the timeout is treated as lost. That is why the fence comes first.
Related: VM migration, kcore SAN, cluster updates, and kcore Vault for the copy that does not live on the cluster.