My homelab's first real outage, and why the fix took 5 minutes

Andrea Straforini — Thu, 22 Oct 2026 00:00:00 +0000

My homelab's first serious hardware failure: a disk that died slowly and quietly under a Kubernetes worker. It's the outage I'd been preparing for, and it felt great to see all that SRE work pay off. 🎉

Summary

Date: 2026-10-22 (all times CEST)</li>
Duration: 22:03–23:05</li>
Root cause: bad sectors on the USB hard drive behind worker-node2's Longhorn disk</li>
Fix: 5 minutes, from the disk swap at 22:30 to services back at 22:35</li>
Data loss: none</li> </ul> Impact #</a> </h2>
What</th> Down</th> Effect</th></tr></thead>

worker-node2</td> 22:03–22:31, 22:51–22:58</td> Paused in `io-error</code>; later a network drop</td></tr>`
Apps on node2 (Jellyfin, *arr, …)</td> 22:03–22:35, 22:51–23:05</td> Stuck Pending: node1 alone lacks the CPU</td></tr>
Postgres standby</td> 22:03–22:47</td> Only replica lost; rebuilt from the primary</td></tr>
Cloudflare tunnel (LXC)</td> 22:30–22:55</td> Disk on the same drive; recreated</td></tr>
Longhorn volumes</td> 22:03–23:05</td> Degraded, served from node1's copies</td></tr> </tbody></table>
Background
`Timeline`#</a> </h2> Time</th> Event</th></tr></thead> 09-05</td> First failed Proxmox backup of a container on the same drive. No alert.</td></tr> 09-12</td> First failed Longhorn backup of a volume on node2. No alert.</td></tr> about 18:00</td> node2 logs medium errors; ext4 remounts a volume read-only.</td></tr> 22:03</td> node2 stops responding: NotReady</code>, VM paused in io-error</code>.</td></tr> 22:30</td> Disk swapped for a 1 TB SSD; node2 back at 22:31.</td></tr> 22:35</td> Longhorn rebuilds replicas onto the new disk.</td></tr> 22:47</td> CloudNativePG rebuilds the standby from the primary.</td></tr> 22:51</td> pve2's Ethernet cable gets knocked while unplugging the old drive; node2 drops out, VM restarted.</td></tr> 23:05</td> All volumes healthy, all pods running.</td></tr> </tbody></table> Root cause#</a> </h2> The USB drive developed unreadable sectors: critical medium error, dev sdb, sector 61086440 op 0x0:(READ)</code></pre> Longhorn saw these as no data available</code>. Once enough reads failed, QEMU paused the VM and Kubernetes lost the node. Contributing factors: Silent backups. Bad sectors broke backups 40 days before they broke the node, and nothing alerted.</li> Replicas couldn't heal. Rebuilds onto node1 read from node2 and failed, so the second copy never existed.</li> A USB spinning disk under the storage layer.</li> </ul> Detection#</a> </h2> Me, noticing Jellyfin was down. Nothing alerted: not the node going NotReady</code>, and none of the earlier signals. Resolution#</a> </h2> Swap the disk, keep every name#</a> </h3> The SSD got the old drive's exact layout, so IaC, Proxmox, fstab and Longhorn needed no changes: qm stop 202 qm set 202 --delete scsi2 # detach the old data disk umount /mnt/dirstore vgchange -an external-lvm vgrename external-lvm external-lvm-old # keep the old drive, don't wipe it pvcreate /dev/sdc vgcreate external-lvm /dev/sdc # same VG name on the SSD lvcreate -L 128G -n dirstore external-lvm mkfs.ext4 -q /dev/external-lvm/dirstore lvcreate -l 99%FREE --thinpool data external-lvm mount /mnt/dirstore qm set 202 --scsi2 external-lvm:128,iothread=1 qm start 202</code></pre>Give Longhorn the old disk's identity#</a> </h3> Longhorn tracks a disk by longhorn-disk.cfg</code> at its root, so it kept waiting for the old one, and refused to remove it while stale replicas were listed. After deleting the dead replica records: echo '{"diskName":"default-disk-23c2c7425e81b15d","diskUUID":"<old disk UUID>","diskDriver":"","state":"ready"}' \ | sudo tee /mnt/data/longhorn-disk.cfg</code></pre> Longhorn saw an empty, ready disk and rebuilt every replica from node1. Not an official procedure, but fine on an empty disk. Let CloudNativePG heal Postgres#</a> </h3> Deleting the dead standby's PVC and pod was enough; CNPG cloned a new one from the primary. What went well#</a> </h2> No data lost on any two-replica volume.</li> CloudNativePG healed itself.</li> Same names meant zero IaC changes, and the fix took 5 minutes.</li> </ul> What went wrong#</a> </h2> 40 days of failing backups, and nobody listening.</li> A replica stuck rebuilding looked like redundancy.</li> Knocking pve2's Ethernet cable while unplugging the old drive caused a second, short outage.</li> </ul> Take home#</a> </h2> Page on a NotReady</code> node. A worker gone for more than a few minutes should reach my phone, instead of waiting for me to notice Jellyfin is down.</li> Alert on failed backups. A Longhorn or Proxmox backup that fails, or hasn't succeeded in a day, should page me. This alone would have caught the disk 40 days early.</li> Alert on replicas stuck rebuilding. A replica that never becomes healthy means the source is broken, not that redundancy is on its way.</li> </ul>

Strafo - cloudnativepg

My homelab's first real outage, and why the fix took 5 minutes

Summary#</span></a> </h2> Date:</strong> 2026-10-22 (all times CEST)</li> Duration:</strong> 22:03–23:05</li> Root cause:</strong> bad sectors on the USB hard drive behind worker-node2's Longhorn disk</li> Fix:</strong> 5 minutes, from the disk swap at 22:30 to services back at 22:35</li>