VM migration

Cold drain and live migration require kcore SAN (storage_backend: ceph). Local filesystem / LVM / ZFS disks stay on one node. See kcore SAN (Ceph).

Shared RBD means the guest disk is not copied when a VM changes host. You choose cold reassignment (stop/start via Nix) or Cloud Hypervisor live migration (memory over TCP while both nodes map the same image).

Cold vs live

Mode	Command	Downtime	When to use
Cold	`kctl drain node …`	Guest stops on source and starts on destination	Evacuate a node for maintenance; fallback if live fails
Live	`kctl migrate vm …`	Brief pause during hypervisor cutover	Keep the workload running while changing host

Prerequisites

Healthy CephCluster; both nodes are members.
VM created with --storage-backend ceph.
Live: allow TCP 18000–18127 between node IPs for the migration stream (same fabric as Ceph public/client is fine).
Roles: vm-admin for migrate vm; cluster-admin for drain and migrate reset-session.

Live migrate a VM

kctl migrate vm app-1 --target-node node-b

# If live fails, fall back to cold reassignment of that VM:
kctl migrate vm app-1 --target-node node-b --allow-cold-fallback

The controller prepares an empty Cloud Hypervisor on the destination (shared RBD already mapped), streams guest memory from the source over TCP, updates which node owns the VM, pushes Nix, then has systemd adopt the migrated process on the destination and clean up the source.

Success responses report mode=live (or mode=cold when fallback ran).

Cold drain a node

kctl drain node node-a
kctl drain node node-a --target-node node-b

Every VM on the source is reassigned in the control plane. Nix push removes units on the source and creates them on destinations. Ceph disks are remapped — not copied. Guests experience a full stop/start.

What happens under the hood

Live

Destination maps the RBD image and starts Cloud Hypervisor with only an API socket.
Destination listens with vm.receive-migration; source calls vm.send-migration to tcp:<dest-ip>:<port>.
Controller moves node_id, pushes config; destination systemd adopts the running process via a live-migrated marker.
Source unit is stopped and the RBD mapping unmapped.

Cold

Controller reassigns node_id (volume row unchanged).
Source Nix no longer defines the VM → unit stops.
Destination Nix defines the VM → map RBD and start Cloud Hypervisor.

Contributor-level detail (RPC names, systemd handoff, RBD features): docs/vm-migration.md in the kcore repository.

Limits

Not supported for filesystem, LVM, or ZFS-backed VMs across nodes.
Live migration takes a destination port from the fixed 18000–18127 range; open host firewalls between members.
Drain does not yet prefer live migrate per VM — use kctl migrate vm when you need live.
A VM with a GPU moves cold onto a free card of the same family. See GPU passthrough.

Troubleshooting

Migration keeps failing with ALREADY_EXISTS

A migration that worked before starts failing immediately with AlreadyExists, naming the destination node. Nothing is actually receiving. The destination still holds the receive session it created when the migration began, because the controller died before it could abort it, and every later migration of that VM to that node is refused until an operator clears the session.

There is no automatic reaping, and that is deliberate. Deciding from the outside that a session is dead is a guess, and getting it wrong tears down a migration that is still running. kcore reports the evidence instead and leaves the call to you.

# 1. Inspect. Read-only; covers every CephCluster member.
kctl get migrate-session app-1

# 2. Dry run: prints what it would clear, clears nothing, exits non-zero.
kctl migrate reset-session app-1 --node node-b

# 3. Clear it.
kctl migrate reset-session app-1 --node node-b --force

Per node, the inspect step reports whether a session is tracked, the reserved port and whether anything is listening on it, the receive VMM's pid and whether it is alive and really belongs to this VM (a recycled pid is reported as recycled, not as live), whether the handoff marker and API socket exist, and a verdict. A node that cannot be reached is reported rather than failing the whole report. Inspecting needs read-only; clearing needs cluster-admin.

Clearing releases the session and its port reservation, kills the receive VMM, removes the marker, pid and API socket files, and unmaps the destination's RBD image. It is idempotent, so clearing a node with no session is a success. --node is mandatory, and the controller refuses the node that currently owns the VM. Once the session is gone, retry the migration as normal.

Read the verdict before you pass --force. Clearing a session whose receive VMM is still alive kills an in-flight migration.