My homelab only pings me when something actually changes

Most monitoring fails the same way. You wire up a check, it runs every few minutes, and it messages you every time it runs. Within a week the messages are wallpaper. You stop reading them, your eyes slide right past the green, and then the one alert that actually mattered scrolls by in the middle of forty that did not. A notification you have trained yourself to ignore is worse than no notification, because you think you are covered.

I learned this the hard way years ago with a tool I built that cheerfully posted to Slack every five minutes, including all the times nothing was wrong. So when I set up a watcher for the homelab, I built it around the opposite idea. It only pings me when something changes.

It runs every half hour and checks a handful of things I care about. But it does not tell me the results. It tells me the transitions. When something crosses a line it sends one alert, and when it comes back it sends one all-clear. In between, silence. Silence is the signal that everything is fine, which means I can actually trust the quiet instead of wondering whether the checker died too.

The trick that makes that work is a small bit of state. The watcher remembers the last status of each thing it tracks, so it can tell a real change from another identical reading. A container that has been down for three hours does not re-alert me six times. It alerted me once, when it went down, and it will tell me once more when it is back. Without that memory you are right back to wallpaper.

It watches a small list: a couple of containers that should always be up, a weekly backup I expect to see happen, a stock price I keep half an eye on, and a flag for an important email landing in a folder I care about. The list is a plain text file. Adding a new watch is a line, and there is nothing to restart.

A couple of details took a second pass to get right. Quiet hours were the first. I do not want a non-urgent blip waking me at 3am, so overnight it holds anything that is not marked urgent and delivers it in the morning, while genuinely urgent items still come straight through. The second was the weekly backup. A backup that should run every week and does not is a real problem, and I want to be nagged about it again the next week if it still has not run, but I do not want a reminder every thirty minutes in between. A plain "did the status change" check would go quiet after the first alert, because the status stayed bad. So the change it watches for there includes which week is due, which lets a fresh miss re-alert without the in-between spam.

The result is a phone that stays quiet, and a quiet that I believe. When it does buzz, it means something genuinely flipped, and I go look, usually by opening TerminalNexus and dropping onto the box to see what happened. No news really is good news here, because I built the thing so that no news is the only thing silence can mean.

Thanks for reading.