Show examples as

VM migration

Cold drain and live migration require kcore SAN (storage_backend: ceph). Local filesystem / LVM / ZFS disks stay on one node. See kcore SAN (Ceph).

Leaving vCenter or Proxmox is a different task. See Import from VMware or Proxmox.

Shared RBD means the guest disk is not copied when a VM changes host. You choose cold reassignment (stop/start via Nix) or Cloud Hypervisor live migration (memory over TCP while both nodes map the same image).

Cold vs live

Mode Command Downtime When to use
Cold kctl drain node … Guest stops on source and starts on destination Evacuate a node for maintenance; fallback if live fails
Live kctl migrate vm … Brief pause during hypervisor cutover Keep the workload running while changing host

Prerequisites

Live migrate a VM

kctl migrate vm app-1 --target-node node-b

# If live fails, fall back to cold reassignment of that VM:
kctl migrate vm app-1 --target-node node-b --allow-cold-fallback

Provider 0.8.0 has no migrate or drain resource. Those stay kctl commands. A Ceph data disk is kcore_volume with storage_class ceph. kcore_vm storage_backend is filesystem, lvm, or zfs.

resource "kcore_volume" "app" {
  name          = "app-data"
  size_bytes    = 107374182400
  storage_class = "ceph"
}

The controller prepares an empty Cloud Hypervisor on the destination (shared RBD already mapped), streams guest memory from the source over TCP, updates which node owns the VM, pushes Nix, then has systemd adopt the migrated process on the destination and clean up the source.

Success responses report mode=live (or mode=cold when fallback ran).

Cold drain a node

kctl drain node node-a
kctl drain node node-a --target-node node-b

Every VM on the source is reassigned in the control plane. Nix push removes units on the source and creates them on destinations. Ceph disks are remapped — not copied. Guests experience a full stop/start.

Automatic failover

When a node misses heartbeats for failover.nodeTimeoutSecs (default 30 seconds, checked every detectIntervalSecs, default 5), the controller marks it not-ready and moves each protected VM to another host. A Ceph guest is fenced first: the controller blocklists the lost host’s clients on the image and its address range through a healthy Ceph member, and only then restarts the guest, in priority order. A ZFS guest with a current replica restarts on its replica node. Other local disks stay on the failed node.

When the node heartbeats again, the controller pushes a configuration that no longer starts the moved guests, so a returning node does not boot a second copy. Set failover.enabled: false in controller.yaml to leave every VM where it is. The full story, with restart priority, anti-affinity, admission control, and the isolation response, is in High availability for VMs.

What happens under the hood

Live

  1. Destination maps the RBD image and starts Cloud Hypervisor with only an API socket.
  2. Destination listens with vm.receive-migration; source calls vm.send-migration to tcp:<dest-ip>:<port>.
  3. Controller moves node_id, pushes config; destination systemd adopts the running process via a live-migrated marker.
  4. Source unit is stopped and the RBD mapping unmapped.

Cold

  1. Controller reassigns node_id (volume row unchanged).
  2. Source Nix no longer defines the VM → unit stops.
  3. Destination Nix defines the VM → map RBD and start Cloud Hypervisor.

Contributor-level detail (RPC names, systemd handoff, RBD features): docs/vm-migration.md in the kcore repository.

Limits

Troubleshooting

Migration keeps failing with ALREADY_EXISTS

A migration that worked before starts failing immediately with AlreadyExists, naming the destination node. Nothing is actually receiving. The destination still holds the receive session it created when the migration began, because the controller died before it could abort it, and every later migration of that VM to that node is refused until an operator clears the session.

There is no automatic reaping, and that is deliberate. Deciding from the outside that a session is dead is a guess, and getting it wrong tears down a migration that is still running. kcore reports the evidence instead and leaves the call to you.

# 1. Inspect. Read-only; covers every CephCluster member.
kctl get migrate-session app-1

# 2. Dry run: prints what it would clear, clears nothing, exits non-zero.
kctl migrate reset-session app-1 --node node-b

# 3. Clear it.
kctl migrate reset-session app-1 --node node-b --force

Provider 0.8.0 has no migrate or drain resource. Those stay kctl commands. A Ceph data disk is kcore_volume with storage_class ceph. kcore_vm storage_backend is filesystem, lvm, or zfs.

Per node, the inspect step reports whether a session is tracked, the reserved port and whether anything is listening on it, the receive VMM's pid and whether it is alive and really belongs to this VM (a recycled pid is reported as recycled, not as live), whether the handoff marker and API socket exist, and a verdict. A node that cannot be reached is reported rather than failing the whole report. Inspecting needs read-only; clearing needs cluster-admin.

Clearing releases the session and its port reservation, kills the receive VMM, removes the marker, pid and API socket files, and unmaps the destination's RBD image. It is idempotent, so clearing a node with no session is a success. --node is mandatory, and the controller refuses the node that currently owns the VM. Once the session is gone, retry the migration as normal.

Read the verdict before you pass --force. Clearing a session whose receive VMM is still alive kills an in-flight migration.