Ransomware resistance
Ransomware that reaches a hypervisor encrypts the guests and then looks for the backups. This page says what kcore does so the backups are still there, what it does not do, and what you have to set up yourself. The backup half is kcore Vault. The hypervisor half is the way kcore nodes are built and updated.
The attacker you are planning for
Someone has an operator password, a stolen client certificate, or root on a hypervisor node. They want to delete or overwrite every copy of your data before they demand money. A backup that the same credentials can reach is not a backup.
Vault: what stops a delete
| Control | What it does |
|---|---|
| Separate machine | Vault runs on its own appliance, not on a hypervisor node. Its disks, accounts, and OS are not the cluster’s. |
| Pull only | The mover on a node serves extents read-only. The appliance pulls. A node holds no credentials for the repository and cannot push, overwrite, or delete a backup. |
| Retention only moves forward | Every stored object has a retain_until time. A later job can extend it. Nothing can shorten it. Delete before that time returns still locked. |
| Object Lock offsite | An S3 bucket with Object Lock takes each object in COMPLIANCE mode. The bucket itself refuses delete and overwrite until the date, including from the account that wrote the object. This is the grade called storage-enforced. |
| Encrypted and signed | Data is chunked, encrypted, and packed. The manifest is signed. A tampered pack or manifest fails verify instead of restoring bad data. |
| Two-person approval | Deleting a target, lowering retention, installing a licence, and changing a role wait for a security officer, a different role from the person asking. |
| Roles and TOTP | Viewer, backup-operator, restore-operator, admin, security-officer. Admin and restore roles must enrol TOTP before the console opens. |
| Own credentials | The coordinator reaches the cluster with a certificate whose role is backup-operator: it can snapshot and read a volume, not manage the cluster. A cluster admin does not get Vault access from that. |
| Verify and restore drills | verify create --full reads objects back and checks them. A restore writes a volume back through the cluster API to Ceph, LVM, or ZFS. Practise it. |
Grades are honest
The console shows a grade per target. appliance-enforced (a ZFS dataset on the appliance) is enforced by the coordinator software. Someone with root on that appliance can delete the files. storage-enforced (Object Lock) is enforced by the bucket. none (S3 without Object Lock) is a cache. For a copy that survives a root compromise of the appliance, replicate to a locked bucket:
kcore-vault target apply offsite --backend s3 --location s3://vault-backups/prod
kcore-vault replicate create --source local --dest offsite
kcore-vault catalog import --target offsite
Create the bucket with Object Lock turned on before the first upload. It cannot be switched on later for a bucket that already has objects. The bucket owner’s credentials and who can delete the bucket itself are your problem, so keep them in a different account from the cluster’s.
The hypervisor side
- Immutable nodes. A kcore node is a NixOS system built from a release tag. The root is rebuilt on update and rolled back on a failed boot. There is no package manager to run on the host, and a change to the system is a new generation, not an edit. See Cluster & node upgrades.
- Canary updates. One node takes a release first and soaks while the controller compares it with the rest. A release that makes the canary unhealthy does not go further.
- mTLS everywhere. Controller, node agents, and
kctlauthenticate with certificates from your own CA, which you can rotate and revoke. See Certificates. - Roles and an audit log. A
vm-admincannot rewrite the cluster. The audit log is append-only, with its own page in the dashboard. - Snapshots on Ceph. Volume snapshots and snapshot policies are fast local recovery points. They live in the same cluster, so they are not a defence against someone who controls the cluster. That is what Vault is for.
What you still have to do
- Put Vault on a different network segment and a different admin domain from the hypervisors. Close what you do not use: the coordinator listens on 9191 on all interfaces, and the console on 443.
- Change the installer password on the first login and set
adminPasswordFile. - Keep the repository key material and the licence somewhere outside the appliance. A replicated copy uses the source repository’s encryption keys, so losing the appliance without them means losing the ability to read the offsite copy.
- On a two-disk appliance the default is a stripe. Losing one disk loses the pool. Ask for a mirror, or keep the Object Lock copy.
- Test a restore on a schedule. A backup you have not restored is a guess.
- Keep the cluster certificates’ private keys off laptops that browse the web.
What this does not cover
kcore does not scan guests for malware. It does not stop an attacker who owns your cloud account that holds the locked bucket, or a security officer who approves what they should not. It makes the destructive path longer, noisier, and impossible for a single stolen credential.