Kubernetes

kcore creates a Kubernetes cluster on nodes that are already members of the kcore cluster. The first apply stores the cluster, generates certificates, and deploys kubelet, etcd, and the API server. Put every member in that first manifest.

Addresses on this page are examples. Pick free addresses on your own network.

Create a cluster

Apply needs cluster-admin. kctl prints the plan, including blocking issues, then asks y/n. Answering y creates the cluster. A later apply of the same name updates it. There is no flag-only create. The manifest is the input.

kctl kubernetes apply -f cluster.yaml
kctl kubernetes get lab
kctl kubernetes list
kctl kubernetes health lab

The manifest below is the whole cluster. Apply it with kctl kubernetes apply -f cluster.yaml. kctl apply -f does not accept this kind.

resource "kcore_cluster_update" "os" {
  name                    = "release-0-5-0"
  version                 = "0.5.0"
  flake_ref               = "github:kcorehypervisor/kcore"
  flake_rev               = "<full-git-sha>"
  nixpkgs_rev             = "<nixpkgs-rev>"
  all_nodes               = true
  strategy                = "one-at-a-time"
  max_unavailable         = 1
  batch_size              = 1
  drain_vms               = false
  activation              = "auto"
  update_kind             = "kcore"
  schema_version          = 59
  previous_schema_version = 56
  min_compatible_reader   = 56
  approval                = "manual"
  approve                 = false

  tested_os_pins {
    nixpkgs_rev        = "<nixpkgs-rev>"
    ceph_rev           = "<ceph-rev>"
    ceph_sha256        = "<ceph-sha256>"
    kubernetes_version = "1.37.0"
  }
}

kind: KubernetesCluster
metadata:
  name: lab
spec:
  clusterCidr: 10.244.0.0/16
  serviceCidr: 10.96.0.0/12
  clusterDns: 10.96.0.10
  apiServerEndpoint: 192.168.40.200:6443
  cni: cilium
  cilium:
    mode: native-routing
    mtu: 1500
    tunnel: disabled
    autoDirectNodeRoutes: true
    loadBalancerIpPools:
      - 192.168.40.220/29
  controlPlaneNodes:
    - nodeId: node-105
      advertiseAddress: 192.168.40.105
      iface: enp6s0
    - nodeId: node-107
      advertiseAddress: 192.168.40.107
      iface: enp6s0
  workerNodes:
    - nodeId: node-103
      advertiseAddress: 192.168.40.103
      iface: enp6s0
  highAvailability:
    enabled: true
    vipAddress: 192.168.40.200
    vipInterface: enp6s0
  scheduleOnControlPlane: true
  localProvisioner:
    enabled: true
    storageClassName: local-path
    mode: shared
    volumes:
      - name: kcore-kdata
        nodeId: node-105
        storageClass: lvm
        sizeBytes: 53687091200
        fsType: ext4
        role: filesystem
      - name: kcore-kdata
        nodeId: node-107
        storageClass: lvm
        sizeBytes: 53687091200
        fsType: ext4
      - name: kcore-kdata
        nodeId: node-103
        storageClass: lvm
        sizeBytes: 53687091200
        fsType: ext4

kind: kubernetes-cluster and kind: kubernetes_cluster are accepted. Fields are camelCase.

What create does

• The phase starts as bootstrapping. The orchestrator generates PKI, deploys every member in parallel, waits for etcd quorum and the API, then marks the cluster ready. It checks again every 30 seconds.
• The first entry in controlPlaneNodes is the bootstrap node. The initialCluster field in the YAML is ignored on create.
• Each control-plane node must already exist and be ready. A missing or not-ready worker is a warning.
• API port 6443 keeps the CNI you set. An empty cni is stored as cilium. Any other API port is stored as bridge. The node module accepts only cilium on the primary cluster. A second cluster on the same hosts must not run a second Cilium. The agents would share BPF pins.
• clusterDns defaults to 10.96.0.10. cni defaults to cilium. Each node's iface defaults to enp6s0.
• scheduleOnControlPlane lets control-plane nodes run pods. workerNodes can then be empty.

Fields

API VIP

highAvailability.enabled starts keepalived on each control plane. Point apiServerEndpoint at vipAddress and the API port. vipAddress and vipInterface are both required when high availability is on.

• The instance name is kcore_api. State is BACKUP, with nopreempt, so a restarted node does not take the VIP back.
• virtual_router_id is 40. advert_int is 1. Each member sends a unicast advertisement every second. That advertisement is the heartbeat.
• Priority is the last octet of the node's advertise address, or 100 when that octet is 0 or 255. Peers are the other etcd members. The address is vipAddress/24 on vipInterface.
• The firewall accepts IP protocol 112 and marks it NOTRACK, so VM masquerade does not rewrite the advertisements.

keepalived does not probe TCP 6443. The VIP moves when advertisements stop, which is when keepalived is down. If only kube-apiserver exits and keepalived stays up, the VIP stays on that node.

When the controller has generated a kubeconfig, the control plane writes it to /etc/kubernetes/admin.conf. Use that file with kubectl. The server is the API endpoint, which is the VIP when high availability is on.

Cilium L2 load balancer

The controller installs Cilium on the primary cluster after the API is ready. The control-plane unit kcore-kubernetes-cilium runs cilium install, then applies CiliumLoadBalancerIPPool and CiliumL2AnnouncementPolicy when loadBalancerIpPools is set. Defaults are native routing, MTU 1500, tunnel disabled, and autoDirectNodeRoutes. A second cluster on the same hosts does not get Cilium.

The host does not run kube-proxy. The install sets kube-proxy replacement, Kubernetes IPAM, and the API address from apiServerEndpoint. Native routing uses clusterCidr. L2 announcements are enabled only when loadBalancerIpPools is not empty. Policy interfaces are the node ifaces. An empty iface is enp6s0. The chart is the one shipped with cilium-cli.

cilium install --namespace kube-system
  --set ipam.mode=kubernetes
  --set kubeProxyReplacement=true
  --set bpf.masquerade=true
  --set routingMode=native
  --set autoDirectNodeRoutes=true
  --set ipv4NativeRoutingCIDR=10.244.0.0/16
  --set mtu=1500
  --set l2announcements.enabled=true
  --set k8sServiceHost=192.168.40.200
  --set k8sServicePort=6443
  --set devices[0]=enp6s0

apiVersion: cilium.io/v2
kind: CiliumLoadBalancerIPPool
metadata:
  name: kcore
spec:
  blocks:
    - cidr: 192.168.40.220/29
---
apiVersion: cilium.io/v2alpha1
kind: CiliumL2AnnouncementPolicy
metadata:
  name: kcore
spec:
  interfaces:
    - enp6s0
  loadBalancerIPs: true

The pool and the policy are both named kcore. This example uses 192.168.40.220/29. Pick a range that does not overlap the VIP or node addresses. A service is announced when loadBalancerClass is empty or io.cilium/l2-announcer. interfaces is a regular expression. The controller writes this manifest to /etc/kubernetes/cilium-l2.yaml and applies it.

Volumes, LVM and ZFS

localProvisioner.enabled is required when volumes are listed. Mode empty or shared is the default. Mode lvm is a thin pool and does not serve claims.

Shared mode creates one LVM logical volume or one ZFS zvol on the named node, formats it, and mounts it. storageClass is lvm or zfs, and the node's storage backend must already be that class. sizeBytes must be positive. An empty fsType becomes ext4. The other accepted value is xfs. An empty role becomes filesystem. The default mount path is /mnt/kcore/<cluster>/<name>.

The control plane then applies the Rancher local-path provisioner, image rancher/local-path-provisioner:v0.0.30, from /etc/kubernetes/provisioner.yaml. The oneshot unit is kcore-kubernetes-provisioner. The StorageClass name defaults to local-path. A claim is a directory on that filesystem. The size on the claim is not a quota. To give the directories more room, raise sizeBytes and apply again. Shrinking is rejected. Another volume adds a path to the same StorageClass. It does not create a second StorageClass.

The controller volume name is k8s-<cluster>-<node>-<volume>. The LVM device is /dev/<vg>/<lv>. The ZFS device is /dev/zvol/<pool>/<dataset>.

A ZFS volume uses the same shared provisioner. Set storageClass to zfs on a node whose storage backend is zfs.

      - name: kcore-kdata
        nodeId: node-103
        storageClass: zfs
        sizeBytes: 53687091200
        fsType: ext4

mode: lvm creates one thin pool of the declared size. Every listed node must use the same name, volume group, size, and filesystem. The role is pool. volumeGroup is required. Leave mountPath empty. kcore does not install a provisioner and does not install a CSI driver. Do not create a CSI driver's pool with this API. kcore deletes only volumes it created. An operator who wants one block device per claim installs that driver and points it at a pool the driver owns.

localProvisioner:
  enabled: true
  mode: lvm
  storageClassName: kcore-lvm
  volumes:
    - name: kcore-kdata
      nodeId: node-105
      storageClass: lvm
      sizeBytes: 53687091200
      role: pool
      volumeGroup: vg_kcore

Upgrades

Kubernetes binaries come from pkgs.kubernetes on the host NixOS pin. There is no separate Kubernetes upgrade command and no cordon-and-drain roller. Move the pin with a ClusterUpdate. See cluster and node upgrades.

A second kctl kubernetes apply updates CIDRs, DNS, the API endpoint, CNI, Cilium settings, the VIP, local volumes, and scheduleOnControlPlane. It redeploys node certificates and node configuration. It does not change cluster membership. It does not cordon or drain nodes. Success returns the phase to ready when the previous phase was updating or degraded. A volume or deploy error sets the phase to degraded.

kctl kubernetes scale --add-control-plane node-id:address and --add-workers node-id:address record those members and set the phase to scaling. The interface on that command is fixed to enp6s0. The orchestrator only bootstraps clusters in phase bootstrapping, so a scale does not join the new nodes. If the existing etcd quorum and API are already healthy, the phase is set back to ready without adding those nodes. --remove logs the node id and does not delete the membership row. Put every member in the first manifest.

Commands

kctl kubernetes delete lab removes the cluster row from the controller. --force is accepted. Deleting the row does not by itself turn Kubernetes off on the nodes.

Limits

• Membership is fixed at create. Scale records nodes and does not join or remove them.
• The VIP heartbeat is a keepalived unicast advertisement every second. It is not a check of port 6443.
• Cilium runs only on the primary cluster, the one whose API port is 6443. An empty loadBalancerIpPools list still installs Cilium and does not create an L2 pool.
• Shared volumes are directories on one filesystem. Claim size is not a quota. mode: lvm does not bind claims.
• A cluster whose API port is not 6443 is stored with bridge CNI, not Cilium.
• Terraform provider 0.5.0 has no Kubernetes cluster resource.