Storage day-2 operations

After initial install, you may need to add data disks, expand storage, or reconfigure layouts. kcore supports day-2 disk operations through disko layouts and NixOS configuration.

Management modes

The file /etc/kcore/disko-management-mode on each node controls how disko layouts are managed after the initial install. There are two modes:

To switch modes, write the desired value to the mode file:

echo "controller-managed" | sudo tee /etc/kcore/disko-management-mode

Validate a disko layout

Before applying any layout, validate it with a dry-run. The command parses the Nix expression with nix-instantiate and reports errors without touching disks:

kctl --node 10.0.0.5:9091 node apply-disko -f layout.nix

Dry-run is the default. No --dry-run flag is needed — the command only formats and mounts when you explicitly pass --apply.

Apply a disko layout

When you are confident the layout is correct, apply it:

kctl --node 10.0.0.5:9091 node apply-disko \
  -f layout.nix \
  --apply \
  --timeout-seconds 300

This runs disko --mode format,mount on the target node. The server-side hard cap is 3600 seconds.

Always validate before applying. Formatting is destructive — it erases data on the targeted devices.

Reconcile NixOS configuration

After changing the disko layout you typically need to update the NixOS configuration so the new mounts persist across reboots:

kctl --node 10.0.0.5:9091 node apply-nix -f config.nix

To only write the configuration file without running nixos-rebuild:

kctl --node 10.0.0.5:9091 node apply-nix -f config.nix --no-rebuild

Inventory and health

Check the current disk layout on a node (runs lsblk remotely):

kctl --node 10.0.0.5:9091 node disks

Inspect storage backend and LVM inventory from the controller:

kctl describe node node-ab12cd34

The output includes the storage backend, volume group details (for LVM), pool status (for ZFS), and mount points.

Operational runbook

Preflight checks

• Run lsblk and findmnt to understand the current disk topology.
• For LVM nodes, check vgs and lvs output.
• For ZFS nodes, check zpool status and zfs list.
• Verify no VMs are using the target devices.

Adding a data disk

• Attach the new disk to the node (physically or via hypervisor).
• Write a disko layout fragment that adds the new device.
• Validate: kctl --node <IP>:9091 node apply-disko -f layout.nix
• Apply: kctl --node <IP>:9091 node apply-disko -f layout.nix --apply
• For filesystem backends, the new disk mounts at /var/lib/kcore/volumes1, volumes2, etc.
• Reconcile NixOS config so mounts survive reboot.

General guidance

• Always validate before applying.
• Keep backups of existing disko layouts.
• Schedule disk operations during maintenance windows when possible.

Out of scope

The following operations are not supported by the day-2 tooling:

• In-place OS repartitioning (changing the OS disk layout after install).
• Automatic cross-backend migration (e.g. moving a VM from filesystem to LVM without manual steps).
• Backend conversion without dedicated migration tooling (e.g. converting a node from LVM to ZFS in place).