High availability for VMs

When a host stops heartbeating, the controller fences its storage, then restarts its protected VMs on the surviving hosts. Nobody has to do anything. This page covers what is protected, how fast it happens, the knobs, and the cases HA does not cover.

What is protected

VM storageWhen the host dies
ceph (kcore SAN)Protected. The image is fenced, then the VM restarts on another host.
zfs with a replica nodeProtected, with a stated data-loss window. The VM starts on the replica node from the last common snapshot.
zfs without a replica, lvm, filesystemNot protected. The disk is on the dead host, so the VM stays there until the host comes back or you restore from backup.

The protection level shows on every VM listing: kctl get vms, the dashboard, and the API (ha_protection). You do not find out after the outage.

What happens when a host dies

  1. Detect. Node agents heartbeat every 15 seconds. A host with no heartbeat for nodeTimeoutSecs (default 30) is marked lost. A detector checks every detectIntervalSecs (default 5).
  2. Fence. The controller asks a healthy Ceph member to blocklist the lost host’s clients, per image and by address range. A host that returns cannot write to an image that has moved. Nothing is reassigned until the fence holds.
  3. Choose what restarts. The controller restarts what was running at the last sync, plus anything with autoStart. A VM that was stopped stays stopped.
  4. Order. VMs restart by restartPriority (high, medium, low), then by memory, largest first. Priority decides who gets capacity first when capacity is short.
  5. Start. Each target host gets its configuration once and starts the guests. The event trail records detect, fence, reassign, and apply, with durations.

In a lab run with Ceph at the older 90-second timeout, the guest was booted on its new host about 20 seconds after the fence. With the 30-second default, expect the fence about 30 to 35 seconds after the last heartbeat, plus guest boot time. Measure your own cluster: kctl get ha-events carries failover_duration_seconds for each move.

A returning host cannot start a stale guest

A host that rebooted boots its old NixOS generation, where the moved VM is still an autoStart unit. kcore closes that path two ways.

Isolation response

A host that lost every controller but still holds running guests would keep them with a fenced disk. After ha.isolationTimeoutSecs (default 60, node agent configuration), the node agent powers off protected Ceph guests it can no longer write, but only when Ceph shows the host is fenced or Ceph itself is unreachable. Local-disk VMs are left alone. Set isolationResponse to leaveRunning (controller failover block, and ha block on the node) if your controller link is flakier than your storage network. The controller-side fence protects the image either way.

Admission control

kcore refuses a placement that would leave protected VMs without a surviving host. hostFailuresToTolerate defaults to 1. For the largest hosts, the remaining hosts must have free memory and CPU for their protected VMs, counting overcommit. CreateVm, SetVmDesiredState, MigrateVm, and DrainNode run the check and answer FAILED_PRECONDITION naming the shortfall. A cluster too small to reserve that capacity is exempt. --ignore-admission-control overrides the check, and the override is audited.

kctl get ha            # failover on or off, lease holder, tolerated host failures, headroom
kctl get ha-events     # detected, fenced, reassigned, applied, failed, skipped
kctl get ha-events --node node-3
kctl get cluster-health

Redundant failover

With replicated controllers, one controller holds a failover lease and runs the detector and the mover. The lease is a replicated register, renewed every leaseRenewSecs (default 5). If the holder dies, another controller claims it after expiry and resumes from the last recorded event instead of fencing twice. A controller whose replication has fallen behind steps down.

Configure it

Controller settings live in controller.yaml. These are the defaults.

failover:
  enabled: true
  nodeTimeoutSecs: 30        # at least two heartbeat intervals (30)
  detectIntervalSecs: 5
  isolationResponse: powerOff   # or leaveRunning
  leaseRenewSecs: 5
  admissionControl:
    hostFailuresToTolerate: 1   # 0 to 5; 0 turns it off

Per VM, in a manifest or on kctl create vm:

kind: VM
metadata:
  name: db-1
spec:
  cpu: 8
  memoryBytes: "16G"
  storageBackend: ceph
  storageSizeBytes: "200G"
  restartPriority: high          # --restart-priority
  antiAffinity: db-tier          # --anti-affinity; spreads the group over hosts
  vmMonitoring: restart          # --vm-monitoring off|restart; restart when the guest stops answering
  vmMonitoringSecs: 120          # --vm-monitoring-secs, at least 30

A ZFS VM can keep an asynchronous replica on another node. The listing shows it as an asynchronous replica instead of unprotected.

kctl create vm app-1 --storage-backend zfs \
  --replica-node node-2 --replica-interval 300 ...   # 60 s to 1 day

Webhooks fire for node.lost, vm.failover.started, vm.failover.completed, and vm.failover.failed, and vm.migrated carries mode: failover.

What HA does not do

Related: VM migration, kcore SAN, cluster updates, and kcore Vault for the copy that does not live on the cluster.