VM migration
Cold drain and live migration require kcore SAN (storage_backend: ceph). Local filesystem / LVM / ZFS disks stay on one node. See kcore SAN (Ceph).
Leaving vCenter or Proxmox is a different task. See Import from VMware or Proxmox.
Shared RBD means the guest disk is not copied when a VM changes host. You choose cold reassignment (stop/start via Nix) or Cloud Hypervisor live migration (memory over TCP while both nodes map the same image).
Cold vs live
| Mode | Command | Downtime | When to use |
|---|---|---|---|
| Cold | kctl drain node … |
Guest stops on source and starts on destination | Evacuate a node for maintenance; fallback if live fails |
| Live | kctl migrate vm … |
Brief pause during hypervisor cutover | Keep the workload running while changing host |
Prerequisites
- Healthy
CephCluster; both nodes are members. - VM created with
--storage-backend ceph. - Live: allow TCP 18000–18127 between node IPs for the migration stream (same fabric as Ceph public/client is fine).
- Roles:
vm-adminformigrate vm;cluster-adminfordrainandmigrate reset-session.
Live migrate a VM
kctl migrate vm app-1 --target-node node-b
# If live fails, fall back to cold reassignment of that VM:
kctl migrate vm app-1 --target-node node-b --allow-cold-fallback
Provider 0.8.0 has no migrate or drain resource. Those stay kctl commands. A Ceph data disk is kcore_volume with storage_class ceph. kcore_vm storage_backend is filesystem, lvm, or zfs.
resource "kcore_volume" "app" {
name = "app-data"
size_bytes = 107374182400
storage_class = "ceph"
}
The controller prepares an empty Cloud Hypervisor on the destination (shared RBD already mapped), streams guest memory from the source over TCP, updates which node owns the VM, pushes Nix, then has systemd adopt the migrated process on the destination and clean up the source.
Success responses report mode=live (or mode=cold when fallback ran).
Cold drain a node
kctl drain node node-a
kctl drain node node-a --target-node node-b
Every VM on the source is reassigned in the control plane. Nix push removes units on the source and creates them on destinations. Ceph disks are remapped — not copied. Guests experience a full stop/start.
Automatic failover
When a node misses heartbeats for failover.nodeTimeoutSecs (default 30 seconds, checked every detectIntervalSecs, default 5), the controller marks it not-ready and moves each protected VM to another host. A Ceph guest is fenced first: the controller blocklists the lost host’s clients on the image and its address range through a healthy Ceph member, and only then restarts the guest, in priority order. A ZFS guest with a current replica restarts on its replica node. Other local disks stay on the failed node.
When the node heartbeats again, the controller pushes a configuration that no longer starts the moved guests, so a returning node does not boot a second copy. Set failover.enabled: false in controller.yaml to leave every VM where it is. The full story, with restart priority, anti-affinity, admission control, and the isolation response, is in High availability for VMs.
What happens under the hood
Live
- Destination maps the RBD image and starts Cloud Hypervisor with only an API socket.
- Destination listens with
vm.receive-migration; source callsvm.send-migrationtotcp:<dest-ip>:<port>. - Controller moves
node_id, pushes config; destination systemd adopts the running process via a live-migrated marker. - Source unit is stopped and the RBD mapping unmapped.
Cold
- Controller reassigns
node_id(volume row unchanged). - Source Nix no longer defines the VM → unit stops.
- Destination Nix defines the VM → map RBD and start Cloud Hypervisor.
Contributor-level detail (RPC names, systemd handoff, RBD features): docs/vm-migration.md in the kcore repository.
Limits
- Not supported for filesystem, LVM, or ZFS-backed VMs across nodes.
- Live migration takes a destination port from the fixed 18000–18127 range; open host firewalls between members.
- Drain does not yet prefer live migrate per VM — use
kctl migrate vmwhen you need live. - A VM with a GPU moves cold onto a free card of the same family. See GPU passthrough.
Troubleshooting
Migration keeps failing with ALREADY_EXISTS
A migration that worked before starts failing immediately with AlreadyExists, naming the destination node. Nothing is actually receiving. The destination still holds the receive session it created when the migration began, because the controller died before it could abort it, and every later migration of that VM to that node is refused until an operator clears the session.
There is no automatic reaping, and that is deliberate. Deciding from the outside that a session is dead is a guess, and getting it wrong tears down a migration that is still running. kcore reports the evidence instead and leaves the call to you.
# 1. Inspect. Read-only; covers every CephCluster member.
kctl get migrate-session app-1
# 2. Dry run: prints what it would clear, clears nothing, exits non-zero.
kctl migrate reset-session app-1 --node node-b
# 3. Clear it.
kctl migrate reset-session app-1 --node node-b --force
Provider 0.8.0 has no migrate or drain resource. Those stay kctl commands. A Ceph data disk is kcore_volume with storage_class ceph. kcore_vm storage_backend is filesystem, lvm, or zfs.
Per node, the inspect step reports whether a session is tracked, the reserved port and whether anything is listening on it, the receive VMM's pid and whether it is alive and really belongs to this VM (a recycled pid is reported as recycled, not as live), whether the handoff marker and API socket exist, and a verdict. A node that cannot be reached is reported rather than failing the whole report. Inspecting needs read-only; clearing needs cluster-admin.
Clearing releases the session and its port reservation, kills the receive VMM, removes the marker, pid and API socket files, and unmaps the destination's RBD image. It is idempotent, so clearing a node with no session is a success. --node is mandatory, and the controller refuses the node that currently owns the VM. Once the session is gone, retry the migration as normal.
Read the verdict before you pass --force. Clearing a session whose receive VMM is still alive kills an in-flight migration.