Add nodes to the cluster

A single-node kcore deployment is useful for evaluation, but production workloads benefit from multiple nodes for scheduling headroom, high-availability controllers, and cross-datacenter replication. This page walks through expanding your cluster from one node to many.

Every install below requires -k (CLI) or insecure: true (YAML) because the live ISO node-agent runs without TLS. After reboot all traffic switches to mTLS automatically.

Add a worker node (agent only)

Boot the new machine from the kcore ISO. Once the live environment is up, install it as an agent-only node pointing to your existing controllers:

kctl node install \
  --node 192.168.40.151:9091 \
  --os-disk /dev/sda \
  --join-controller 192.168.40.107:9090 \
  --join-controller 192.168.40.105:9090 \
  --dc-id DC1 \
  -k

# worker-node.yaml
kind: NodeInstall
metadata:
  name: kvm-node-151
spec:
  node: 192.168.40.151:9091
  osDisk: /dev/sda
  joinControllers:
    - 192.168.40.107:9090
    - 192.168.40.105:9090
  dcId: DC1
  insecure: true

resource "kcore_node" "worker" {
  address         = "192.168.40.151:9091"
  os_disk         = "/dev/sda"
  certs_dir       = kcore_cluster.prod.certs_dir
  join_controller = "192.168.40.107:9090"
  dc_id           = "DC1"
  insecure        = true
  hostname        = "kvm-node-151"
}

You can specify multiple --join-controller flags so the node agent knows about all controllers for failover. This node runs the node agent only—no controller process.

After reboot the node registers with the controller and is auto-approved by default. Verify with:

kctl get nodes

Add a second controller (HA)

For HA you want at least two controllers. The new node starts its own controller and joins the existing one:

kctl node install \
  --node 192.168.40.105:9091 \
  --os-disk /dev/sda \
  --run-controller \
  --join-controller 192.168.40.107:9090 \
  --dc-id DC1 \
  -k

# ha-controller.yaml
kind: NodeInstall
metadata:
  name: kvm-node-105
spec:
  node: 192.168.40.105:9091
  osDisk: /dev/sda
  runController: true
  joinControllers:
    - 192.168.40.107:9090
  dcId: DC1
  insecure: true

resource "kcore_node" "ha" {
  address         = "192.168.40.105:9091"
  os_disk         = "/dev/sda"
  certs_dir       = kcore_cluster.prod.certs_dir
  run_controller  = true
  join_controller = "192.168.40.107:9090"
  dc_id           = "DC1"
  insecure        = true
  hostname        = "kvm-node-105"
}

After reboot, the two controllers automatically discover each other and begin CRDT replication. Your kctl config is auto-updated to include the new controller endpoint, so kctl can fail over transparently:

kctl get nodes

Both controllers will show an identical view of the cluster. You can also verify from a specific controller:

kctl -s 192.168.40.105:9090 get nodes
kctl -s 192.168.40.107:9090 get nodes

Cross-DC: add a node in DC2

Multi-datacenter deployments tag each node with a datacenter identifier:

kctl node install \
  --node <NEW_NODE_IP>:9091 \
  --os-disk /dev/sda \
  --run-controller \
  --join-controller 192.168.40.107:9090 \
  --dc-id DC2 \
  -k

# dc2-controller.yaml
kind: NodeInstall
spec:
  node: <NEW_NODE_IP>:9091
  osDisk: /dev/sda
  runController: true
  joinControllers:
    - 192.168.40.107:9090
  dcId: DC2
  insecure: true

resource "kcore_node" "dc2" {
  address         = "<NEW_NODE_IP>:9091"
  os_disk         = "/dev/sda"
  certs_dir       = kcore_cluster.prod.certs_dir
  run_controller  = true
  join_controller = "192.168.40.107:9090"
  dc_id           = "DC2"
  insecure        = true
}

Resources created on this node are tagged with DC2. Controllers across datacenters replicate state via conflict-free replicated data types (CRDTs) with adaptive cross-DC timeouts, so each site maintains an eventually-consistent view of the cluster.

You can target a specific datacenter when creating VMs:

kctl create vm db-replica --dc DC2 \
  --image <image-url> --network default

resource "kcore_vm" "db_replica" {
  name               = "db-replica"
  cpu                = 2
  memory_bytes       = 4 * 1024 * 1024 * 1024
  storage_backend    = "filesystem"
  storage_size_bytes = 40 * 1024 * 1024 * 1024
  image_url          = "<image-url>"
  image_sha256       = "<sha256>"
  target_dc          = "DC2"

  nic {
    network = "default"
  }
}

Registration and approval

By default, nodes are auto-approved on registration. This means a node with a valid mTLS certificate issued by the cluster CA is immediately available for scheduling after reboot.

If you prefer manual approval, set requireManualApproval: true in the controller configuration. In manual mode, nodes register as pending and must be explicitly approved:

kctl get nodes
kctl node approve <NODE_ID>

If a registration is unexpected or suspicious, reject it:

kctl node reject <NODE_ID>

Even with auto-approval enabled, all node communication is authenticated via mTLS. Only nodes with a certificate signed by the cluster CA can register.

Cordon and removal

kctl node cordon <NODE_ID> marks the node unschedulable. VMs already running there stay. Heartbeats do not clear the cordon. kctl node uncordon <NODE_ID> returns an approved node to scheduling. A node that is draining or drained stays that way; cordon does not evacuate guests. Disk-layout evacuate still uses the separate draining status.

kctl node delete <NODE_ID> removes a node only after it has no VMs, workloads, or networks. It revokes every certificate issued to that node with RFC 5280 reason 5 (cessation of operation), refreshes the revocation set and CRL, then deletes the row. kctl revoke cert --node <NODE_ID> still revokes certificates without deleting the node.

kctl get cluster-health summarizes whether approved nodes are ready, how many VMs still sit on a not-ready node, certificates inside 30 days, and unresolved replication conflicts. kctl backup -o snapshot.sqlite writes a consistent copy of the controller database. kctl restore controller -f snapshot.sqlite --confirm replaces that database. Restart any other controller after a restore so it does not keep serving the previous copy.

Certificates

Node certificates are renewed automatically when within 30 days of expiry. Operators can rotate the sub-CA at any time:

kctl rotate sub-ca

Inspect certificate expiry for all nodes:

kctl get nodes

Controller replication

When a second controller joins the cluster, replication starts automatically:

1. The new controller is configured with the first controller as a replication peer.
2. On first connection, the new controller pulls all existing replication events (node registrations, networks, VMs, etc.).
3. The first controller discovers the new controller via the replication ack handshake and begins polling it in return.
4. Both controllers now maintain a fully converged, eventually-consistent view of the cluster state.

No manual peering configuration is needed. Controller discovery is handled via CRDT-replicated controller.register events.

Quick reference

Optional install flags

Example: 3-node cluster walkthrough

Starting from zero, a full 3-node cluster with 2 controllers takes four manifests (or commands):

# 1. Create cluster PKI (one-time)
kctl create cluster --controller 192.168.40.107:9090 --context prod

# 2. Install first node as controller
kctl node install --node 192.168.40.107:9091 \
  --os-disk /dev/sda --run-controller --dc-id DC1 -k

# 3. Install second node as controller (joins first)
kctl node install --node 192.168.40.105:9091 \
  --os-disk /dev/sda --run-controller \
  --join-controller 192.168.40.107:9090 --dc-id DC1 -k

# 4. Install third node as worker (joins both controllers)
kctl node install --node 192.168.40.151:9091 \
  --os-disk /dev/sda \
  --join-controller 192.168.40.107:9090 \
  --join-controller 192.168.40.105:9090 --dc-id DC1 -k

# 1. cluster.yaml — create cluster PKI (one-time)
kind: Cluster
metadata:
  name: prod
spec:
  controller: 192.168.40.107:9090
---
# 2. node-controller-1.yaml — first controller
kind: NodeInstall
metadata:
  name: kvm-node-107
spec:
  node: 192.168.40.107:9091
  osDisk: /dev/sda
  runController: true
  dcId: DC1
  insecure: true
---
# 3. ha-controller.yaml — second controller
kind: NodeInstall
metadata:
  name: kvm-node-105
spec:
  node: 192.168.40.105:9091
  osDisk: /dev/sda
  runController: true
  joinControllers:
    - 192.168.40.107:9090
  dcId: DC1
  insecure: true
---
# 4. worker.yaml — agent-only node
kind: NodeInstall
metadata:
  name: kvm-node-151
spec:
  node: 192.168.40.151:9091
  osDisk: /dev/sda
  joinControllers:
    - 192.168.40.107:9090
    - 192.168.40.105:9090
  dcId: DC1
  insecure: true

resource "kcore_cluster" "prod" {
  name       = "prod"
  controller = "192.168.40.107:9090"
  certs_dir  = "${path.module}/certs"
}

resource "kcore_node" "controller" {
  bootstrap      = true
  address        = "192.168.40.107:9091"
  os_disk        = "/dev/sda"
  certs_dir      = kcore_cluster.prod.certs_dir
  run_controller = true
  dc_id          = "DC1"
  insecure       = true
  hostname       = "kvm-node-107"
}

resource "kcore_node" "ha" {
  address         = "192.168.40.105:9091"
  os_disk         = "/dev/sda"
  certs_dir       = kcore_cluster.prod.certs_dir
  run_controller  = true
  join_controller = kcore_node.controller.controller_address
  dc_id           = "DC1"
  insecure        = true
  hostname        = "kvm-node-105"
}

resource "kcore_node" "worker" {
  address         = "192.168.40.151:9091"
  os_disk         = "/dev/sda"
  certs_dir       = kcore_cluster.prod.certs_dir
  join_controller = kcore_node.controller.controller_address
  dc_id           = "DC1"
  insecure        = true
  hostname        = "kvm-node-151"
}

After each node reboots, it auto-registers and auto-approves. Controllers auto-discover each other and replicate state via CRDTs. Run kctl get nodes to verify the full cluster.