kcore Vault

kcore Vault 0.1.1 is the backup appliance. It is a separate machine from the hypervisor nodes. This page is the 0.6.1 operator guide: what the dashboard shows, how a backup is built, how immutability is enforced, and how the offsite copy is stored on ZFS or in S3.

Open the console

The console listens on https://<appliance-ip>/ (port 443). Sign in as admin. The first password is the installer password, kcore, unless services.kcore-vault.dashboard.adminPasswordFile sets another. Change it after the first login. Admin and restore roles must enroll TOTP before the console opens.

The header is the same shape as the provisioning dashboard: the word kcore in blue, then Vault. Pages are Overview, Inventory, Policies, Jobs, Restore, Files, Verify, Reports, Licence, and Settings. Job rows, toasts, and the active-job tray update from a server-sent event stream. You do not reload the page to see a phase change.

Roles are viewer, backup-operator, restore-operator, admin, and security-officer. Deleting a target, lowering retention, installing a licence, or changing a role waits for a security officer. OIDC is optional (dashboard.oidcIssuer).

What is installed

The live ISO is not the hypervisor. install-vault erases the OS disk and, when you pass data disks, builds ZFS pool vault.

Two data disks of the same size become one striped pool. The pool on this machine is about 476 GB raw and about 462 GB usable. A stripe uses both disks for capacity. Losing either NVMe disk loses the pool. A mirror would keep one copy of every block and would hold about half of that. The installer does not create the mirror for you.

The dataset mountpoint is legacy. NixOS mounts it from the filesystem table. A native ZFS mountpoint makes systemd fail the boot and drop to emergency mode, which is what the first reboot of this machine did.

Architecture

operator workstation
    browser https://appliance/  -->  console :443
                           |
                           | gRPC
                           v
kcore cluster                         kcore Vault appliance
controller :50051  <-- mTLS --  coordinator :9191
  backup-operator role                 sqlite catalog
  client CN kctl:vault                 engine: chunk, encrypt, sign
node agent                               |
  freeze, snapshot, expose               +-- fs repository
  CBT extents                            |     /var/lib/kcore-vault
  mover :9192  <-- pull -- coordinator   |     ZFS pool vault
                                         |
                                         +-- offsite target
                                               S3 Object Lock
                                               or another fs path

The coordinator talks to the kcore controller with a client certificate. The certificate common name is kctl:vault and the controller role is backup-operator. Set these on the appliance:

export KCORE_CONTROLLER_ENDPOINT=https://controller:50051
export KCORE_CONTROLLER_CA=/etc/kcore/ca.pem
export KCORE_CONTROLLER_CERT=/etc/kcore/vault.crt
export KCORE_CONTROLLER_KEY=/etc/kcore/vault.key

One backup of a VM runs in this order:

1. Optional freeze, so the guest filesystem is quiet.
2. Snapshot the volumes on the cluster.
3. Expose the snapshot on the node that holds the disk.
4. Ask for changed-block extents. The first backup of a volume has no parent, so it is a full read.
5. Pull the bytes. --mover auto uses the mover on that node. Otherwise the coordinator calls ReadVolumeBlocks. Blocks on the wire are 1 MiB. Each request window is 64 MiB, because the node holds that window in memory.
6. Chunk, encrypt, and pack the bytes into the repository. The manifest is signed.
7. Unexpose the snapshot.

The next backup of the same VM stores a parent id. Unchanged chunks are not copied again. The catalogue row points at the parent so a restore can rebuild the volume.

How immutability works

Every stored object has a retain_until time. Vault will not delete it before that time. Retention only moves forward. A later job can extend the date. It cannot shorten it.

There are three grades. The dashboard target table shows the grade for each target.

Pack files, indexes, volume indexes, and the signed backup manifest are the objects. The sqlite file on the appliance is the catalogue. It is not the backup. Deleting a catalogue row does not delete a locked object.

GFS retention is a catalogue mark, not a delete. The reconciler runs every 60 seconds (--reconcile-secs). For each policy it keeps the newest hourly backups inside 48 hours, the newest daily for each day, weekly, monthly, and yearly, using the counts on the policy. Anything outside those sets is marked expired. kcore-vault gc run --target local then deletes pack objects whose retention has passed and which no live backup still references. A live backup keeps every object it names. GC will not remove those.

kcore-vault verify create --full --target local reads the objects back and checks them. A failed verify is a broken backup, not a retention event.

Offsite tier

A target is either fs or s3. Those are the only backends the coordinator accepts.

ZFS, as the fs repository

The appliance store is an fs target whose directory is the ZFS dataset. The default location is /var/lib/kcore-vault. The grade is appliance-enforced. ZFS gives you checksums, compression, and a pool that can span the data disks. It does not give you Object Lock. There is no zfs send job in Vault. If you want a second ZFS copy, mount that dataset on a path and register another fs target:

kcore-vault target apply local --backend fs --location /var/lib/kcore-vault
kcore-vault target apply zfs-offsite --backend fs --location /mnt/offsite-zfs/vault

zfs-offsite is still appliance-enforced. The machine that holds that directory can delete it. Use it when the second copy is another Vault host, or a ZFS dataset you replicate yourself with zfs send outside Vault. Vault will not run that send.

S3 Object Lock

Create the bucket with Object Lock turned on before the first upload. Vault cannot enable Object Lock on a bucket that already has objects. Then:

export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export AWS_ENDPOINT_URL=https://rgw.example:7480
export AWS_REGION=us-east-1
export AWS_S3_FORCE_PATH_STYLE=true

kcore-vault target apply offsite --backend s3 --location s3://vault-backups/prod

AWS_ENDPOINT_URL is optional. Set it for MinIO, Ceph RGW, or any other S3 API. Leave it unset for AWS. Path-style addressing is required for most on-prem gateways. On NixOS the same values are files and an endpoint, not shell exports:

services.kcore-vault.aws = {
  accessKeyIdFile = "/run/secrets/s3-access-key";
  secretAccessKeyFile = "/run/secrets/s3-secret-key";
  endpointUrl = "https://rgw.example:7480";
};

The coordinator reads those files as root and copies them into mode 0600 files owned by the kcore-vault user. Each object is locked until its own retain_until. That date comes from the backup policy, not from a single bucket default. COMPLIANCE is the mode Vault writes. Governance mode is not used, because governance can be removed by a privileged user.

A bucket without Object Lock can still be registered. Pass the CLI flag that grades the target none (the kcore-vault target apply help calls it out as s3 without Object Lock). The dashboard will show grade none. Treat that target as a cache, not as a record.

Copy the local pool to offsite

kcore-vault replicate create --source local --dest offsite
kcore-vault catalog import --target offsite

replicate copies the sealed objects. The destination keeps the source encryption keys, because the repository config object is copied as bytes. catalog import reads the manifests on that target and inserts catalogue rows so the next backup can use them as parents. Run the import after the replicate, on the appliance that should list those backups.

You can replicate fs to fs (local ZFS to another ZFS path) or fs to s3. The destination grade is the destination's grade. A ZFS path stays appliance-enforced. An Object Lock bucket becomes storage-enforced even when the source was only appliance-enforced.

Policies, restore, verify

kcore-vault policy apply nightly --schedule @daily --vm web1 --target local \
  --keep-daily 14 --keep-weekly 8 --keep-monthly 12

The schedule is the reconciler's clock. @daily enqueues a job when the last successful backup for that policy is older than a day. --keep-* is the GFS set described above. The dashboard Policies page writes the same record.

Restore writes a volume back through the cluster API. On the Restore page you name the backup id, the source volume, and the destination. The storage class on that form is where the hypervisor puts the restored disk: ceph, lvm, or zfs. That class is the cluster disk, not the Vault pool. For lvm and zfs you also name the node.

A file restore, used by the CLI against a path, writes the bytes to a file on the appliance. It does not create a VM.

Licence

The licence is a signed JSON document. The fields covered by the Ed25519 signature are customer, expires_unix, limit_tib, limit_vms, and mode. The signature is hex, over the canonical JSON of those fields, checked with KCORE_VAULT_LICENCE_PUBKEY.

With no row in the database the status is missing. The Overview pill is red. After a signature check that passes, and the expiry is still in the future, the status is ok on the dashboard (the API word is valid) and the pill is green. expired is red.

mode is advisory or enforce. Advisory mode records the licence and still accepts backup jobs when the licence is missing or expired. Enforce mode rejects new backup jobs unless the status is valid. Restores are not gated by that check. Set services.kcore-vault.enforceLicence = true on the appliance when a missing licence must stop backups.

Install the document from the Licence page, or with kcore-vault licence install. The text is either JSON or a small set of key = value lines: customer, expires_unix, mode, limit_vms, limit_tib, signature.

Ports on the appliance

The local screen on the appliance is the vault console. It is read-only. Administration is the browser at https://<appliance-ip>/, or kcore-vault over SSH.