kcore SAN (Ceph)

Phases 1–2 (CephCluster fabric + RBD VM disks / cold drain) and live migration over shared RBD ship in kcore.

kcore currently supports local storage backends (filesystem, LVM, ZFS) where VM disks live on the host running the VM. kcore SAN adds distributed shared block storage using Ceph, so VM disks are replicated across nodes and can move with the workload.

Vision

• Distributed block storage across cluster nodes.
• Ceph OSDs on dedicated data disks, managed by a declarative CephCluster resource.
• RBD (RADOS Block Device) volumes serve as VM disks.
• Storage is replicated across nodes for redundancy.

How it integrates

• Phase 1 (shipped): day-2 CephCluster YAML applied with kctl apply — controller reconciler configures NixOS services.ceph, bootstraps MON/MGR/OSD (including ceph-volume on the OSD disk), and tracks cluster health. OSDs are not created via node install --storage-backend.
• Phase 2 (shipped): RBD pool and --storage-backend ceph for VMs; volumes are cluster-scoped and map on any member node; cold drain moves VMs without copying disks.
• Shipped: dedicated Ceph cluster/replication network (e.g. 10Gb fabric) separate from the public/client path.
• Live migration (shipped): Cloud Hypervisor live migration on shared RBD (kctl migrate vm). See VM migration.

Moving VMs (cold and live)

Because the disk is an RBD image in pool kcore-vms, changing hosts does not copy guest blocks. Choose the mode that matches your downtime budget:

• Live: kctl migrate vm <id> --target-node <node> — memory streams over TCP; both nodes map the same image during cutover.
• Cold: kctl drain node <node> — control plane reassigns VMs; Nix stop on source / start on destination.
• Fallback: kctl migrate vm … --allow-cold-fallback tries live, then cold for that VM.

Operator runbook: VM migration. Architecture and RPC flow: docs/vm-migration.md · Ceph integration: docs/ceph.md.

Target use cases

• HA VM storage — survive single-node failure without data loss.
• Live migration — VMs can move between nodes without copying disks (kctl migrate vm).
• Multi-node data redundancy — configurable replication factor (e.g. 2 or 3 copies).
• Shared image cache — VM base images stored once in Ceph, cloned instantly.

How it complements local backends

Local backends (filesystem, LVM, ZFS) remain the right choice for single-node, performance-sensitive workloads where raw I/O throughput matters most. kcore SAN is designed for workloads that prioritise availability over raw I/O performance.

Operators choose per-VM: a database that needs low-latency local NVMe stays on LVM, while an application server uses Ceph-backed storage for easy failover.

Relationship to existing storage classes

kcore SAN appears as a new storage class (ceph) alongside the existing filesystem, lvm, and zfs classes:

$ kctl get storage-class
NAME          NODES
filesystem    3
lvm           2
zfs           1
ceph          3

Nodes that are members of a healthy CephCluster are listed under the ceph storage class. Creating a VM on Ceph storage:

kctl create vm app-1 \
  --storage-backend ceph \
  --storage-size-bytes 107374182400 \
  --image <https-url> \
  --image-sha256 <sha256>

Example CephCluster manifest (Phase 1)

kind: CephCluster
metadata:
  name: lab-san
spec:
  publicNetwork: "10.10.0.0/24"
  clusterNetwork: "10.20.0.0/24"
  replication:
    size: 3
    minSize: 2
  nodes:
    - nodeId: dell-1
      monAddr: "10.10.0.11:6789"
      clusterAddr: "10.20.0.11"
      publicIface: eth1
      clusterIface: eth2
      osdDevice: /dev/nvme0n1

resource "kcore_ceph_cluster" "lab_san" {
  name            = "lab-san"
  public_network  = "10.10.0.0/24"
  cluster_network = "10.20.0.0/24"
  size            = 3
  min_size        = 2

  node {
    node_id       = "dell-1"
    mon_addr      = "10.10.0.11:6789"
    cluster_addr  = "10.20.0.11"
    public_iface  = "eth1"
    cluster_iface = "eth2"
    osd_device    = "/dev/nvme0n1"
  }
}

resource "kcore_volume" "app" {
  name          = "app-data"
  size_bytes    = 107374182400
  storage_class = "ceph"
}

resource "kcore_shared_filesystem" "home" {
  name         = "home"
  ceph_cluster = kcore_ceph_cluster.lab_san.name
  fs_name      = "home"

  client {
    name  = "web"
    paths = ["/"]
  }
}

resource "kcore_object_store" "s3" {
  name         = "s3"
  ceph_cluster = kcore_ceph_cluster.lab_san.name
  port         = 7480
}

resource "kcore_object_user" "app" {
  name  = "app"
  store = kcore_object_store.s3.name
}