Extended outage - Ongoing till I get it fixed :(
Things were going well with the maintenance. Till they were not going at all.
Bright side: all your user accounts are still good and will be when things come back. There shouldn't be a need to create new accounts for anyone. Keep your eyes peeled here to see when it does come back up!
Everything was going along wonderfully. The software update on the NAS went smoothly and by the numbers. I moved on to adding the "last" hard drive to the existing array. Turned off the server, slotted & connected the new drive, then fired it back up.
 WHIRR-BUZZ-CLICK
 "Hmm, it's not supposed to take that long. I hope that everything is okay."
Spoiler: everything was not okay.
20 minutes later, I knew that something was REALLY wrong when the drive software told me that two (TWO!!1!) of the Seagate drives in my RAID array had failed. I could handle one (hell, I half expected it), but not two. Two dead drives meant that ALL the data on the array was gone. 29TB (give or take) of media has evaporated, along with some personal files and my partners portfolio. I did take some steps to make offsite backups of those last two, but the cost of backing up 29-ish TB of files in the cloud means that those have no backup and must be acquired again. Doable, but it's gonna take some time.
I'm going to take this time to rebuild things more properly, this time with some more cooling and much more regular maintenance. Doing some drive testing once in a while would have revealed the failed drives much earlier, before they became a compound problem and took all the media with them. Expect some more regular downtime when things do come back. I'm also going to take this opportunity to reinstall the underlying OS with something that makes a little more sense (not ubuntu anymore) and maybe even embracing ZFS. Maybe it's time to try TrueNAS Scale or OpenMediaVault? We'll see.
Keep watching here to get updates on when this is all running again.
- Stickus