My homelab's first real outage, and why the fix took 5 minutes
My homelab's first serious hardware failure: a disk that died slowly and quietly under a Kubernetes worker. It's the outage I'd been preparing for, and it felt great to see all that SRE work pay off. 🎉
Summary
- Date: 2026-10-22 (all times CEST)
- Duration: 22:03–23:05
- Root cause: bad sectors on the USB hard drive behind worker-node2's Longhorn disk
- Fix: 5 minutes, from the disk swap at 22:30 to services back at 22:35
- Data loss: none
Impact
| What | Down | Effect |
|---|---|---|
| worker-node2 | 22:03–22:31, 22:51–22:58 | Paused in io-error; later a network drop |
| Apps on node2 (Jellyfin, *arr, …) | 22:03–22:35, 22:51–23:05 | Stuck Pending: node1 alone lacks the CPU |
| Postgres standby | 22:03–22:47 | Only replica lost; rebuilt from the primary |
| Cloudflare tunnel (LXC) | 22:30–22:55 | Disk on the same drive; recreated |
| Longhorn volumes | 22:03–23:05 | Degraded, served from node1's copies |
Background
Each Kubernetes worker is a Proxmox VM with a second, 128 GB disk at /mnt/data for Longhorn replicas. Most volumes keep one replica per worker; CloudNativePG volumes keep one in total, since CNPG replicates itself. worker-node2's data disk sat on external-lvm, a thin pool on a 1 TB 2.5" USB hard drive.
Timeline
| Time | Event |
|---|---|
| 09-05 | First failed Proxmox backup of a container on the same drive. No alert. |
| 09-12 | First failed Longhorn backup of a volume on node2. No alert. |
| about 18:00 | node2 logs medium errors; ext4 remounts a volume read-only. |
| 22:03 | node2 stops responding: NotReady, VM paused in io-error. |
| 22:30 | Disk swapped for a 1 TB SSD; node2 back at 22:31. |
| 22:35 | Longhorn rebuilds replicas onto the new disk. |
| 22:47 | CloudNativePG rebuilds the standby from the primary. |
| 22:51 | pve2's Ethernet cable gets knocked while unplugging the old drive; node2 drops out, VM restarted. |
| 23:05 | All volumes healthy, all pods running. |
Root cause
The USB drive developed unreadable sectors:
critical medium error, dev sdb, sector 61086440 op 0x0:(READ)Longhorn saw these as no data available. Once enough reads failed, QEMU paused the VM and Kubernetes lost the node.
Contributing factors:
- Silent backups. Bad sectors broke backups 40 days before they broke the node, and nothing alerted.
- Replicas couldn't heal. Rebuilds onto node1 read from node2 and failed, so the second copy never existed.
- A USB spinning disk under the storage layer.
Detection
Me, noticing Jellyfin was down. Nothing alerted: not the node going NotReady, and none of the earlier signals.
Resolution
Swap the disk, keep every name
The SSD got the old drive's exact layout, so IaC, Proxmox, fstab and Longhorn needed no changes:
qm stop 202
qm set 202 --delete scsi2 # detach the old data disk
umount /mnt/dirstore
vgchange -an external-lvm
vgrename external-lvm external-lvm-old # keep the old drive, don't wipe it
pvcreate /dev/sdc
vgcreate external-lvm /dev/sdc # same VG name on the SSD
lvcreate -L 128G -n dirstore external-lvm
mkfs.ext4 -q /dev/external-lvm/dirstore
lvcreate -l 99%FREE --thinpool data external-lvm
mount /mnt/dirstore
qm set 202 --scsi2 external-lvm:128,iothread=1
qm start 202Give Longhorn the old disk's identity
Longhorn tracks a disk by longhorn-disk.cfg at its root, so it kept waiting for the old one, and refused to remove it while stale replicas were listed. After deleting the dead replica records:
echo '{"diskName":"default-disk-23c2c7425e81b15d","diskUUID":"<old disk UUID>","diskDriver":"","state":"ready"}' \
| sudo tee /mnt/data/longhorn-disk.cfgLonghorn saw an empty, ready disk and rebuilt every replica from node1. Not an official procedure, but fine on an empty disk.
Let CloudNativePG heal Postgres
Deleting the dead standby's PVC and pod was enough; CNPG cloned a new one from the primary.
What went well
- No data lost on any two-replica volume.
- CloudNativePG healed itself.
- Same names meant zero IaC changes, and the fix took 5 minutes.
What went wrong
- 40 days of failing backups, and nobody listening.
- A replica stuck rebuilding looked like redundancy.
- Knocking pve2's Ethernet cable while unplugging the old drive caused a second, short outage.
Take home
- Page on a
NotReadynode. A worker gone for more than a few minutes should reach my phone, instead of waiting for me to notice Jellyfin is down. - Alert on failed backups. A Longhorn or Proxmox backup that fails, or hasn't succeeded in a day, should page me. This alone would have caught the disk 40 days early.
- Alert on replicas stuck rebuilding. A replica that never becomes healthy means the source is broken, not that redundancy is on its way.