A homelab that heals itself at 4am

The classic homelab failure goes like this. Something falls over at two in the morning. A container exits and does not come back, or a timer stops firing. Nothing is on fire, so nothing wakes you. You find out at nine, when you go to use the thing and it is not there. The gap between "it broke" and "I noticed" is where all the annoyance lives.

So I built something to close that gap. Every day at 4am a job wakes up, checks every system on the box, fixes what is safe to fix on its own, and leaves me a single message about the rest.

The interesting part was deciding what "safe to fix" means, because a robot with a hammer loose on your server at 4am is a worse problem than the one it is trying to solve. The rule I landed on is narrow on purpose. It will restart a timer or a service that has stopped. It will bring a container back up if it has fallen over. It will re-run an indexer that is safe to run twice. That is the whole list. The big stateful things, the databases and the heavy containers, it will not touch. It only reports those and waits for me. Under-fix and tell, never guess. I would rather it leave ten things for me than confidently break one.

There is no AI in this one, and that is the point. It is a plain deterministic checker. No model, no tokens, no creativity. For a question like "is this timer active and did its last run exit clean," you want something boring and exact that does the same thing every night, not something clever that might decide to be interesting. Cleverness is for problems that need it. This is not one.

The message it sends is short. How many systems are healthy, what it fixed while I slept, and what is still broken and needs me. If everything is fine, it tells me that and nothing more. No news is good news, and the message reflects that.

My favorite piece is the part that does not trust its own watch list. It would be easy to write a checker that only looks at the things you remembered to tell it about, and then quietly never notices the service you spun up last week and forgot to add. So every night, before it reports, it lists what is actually running on the box right now and compares that against what it is watching. Anything running that nothing is watching gets flagged as "not watched yet" until I either add it or tell the job to ignore it. New things I build cannot silently fall outside the net, which is exactly the kind of mistake I make.

Changing what it watches and what it is allowed to heal is a text file, not code. One line per system, with a flag that says whether the job may fix it or only report it. Adding a new service to the morning round is a line, and I never have to open the script.

The first night it ran, it watched a few dozen systems and found almost everything healthy, plus a couple of things genuinely down that a restart would not bring back. It did the right thing there: it did not flail at them, it just wrote them up and left them for me. That is the behavior I wanted. When I do sit down to deal with whatever it could not, I am in TerminalNexus and SSH'd into the box in a couple of keystrokes, working from its list instead of going to hunt for the problem myself.

Thanks for reading. If your homelab has a 2am habit of breaking quietly, a small nightly pass that fixes the safe stuff and tattles on the rest is worth the afternoon it takes to build. Tell me what you would and would not let it touch.