The Daemon That Died for Two Weeks, and Nobody Noticed

One day I opened a page that shows me product prices, and noticed the prices were the same set as several days back. At first I figured the market was just quiet. Then I looked underneath and found the thing that fetches those prices had stopped. The page still loaded, the numbers were all there. They were just numbers frozen in place.

That price-fetcher is a daemon. It's a program I set to stay running, doing its own work in the background all the time, without me telling it to each time. But with nobody watching, the day it went down, no sound went up to say so.

The scary part of a daemon isn't when it crashes. It's when it crashes quietly. The surface still looks fine, and days go by before you realize, sometimes weeks. So this article is about what a daemon is, told through the ones that died quietly in our own house. Three parts: first what a daemon actually is, then why they die so silently, and finally how to make one make a sound when it goes.

Part 1What a daemon actually is

The programs we use day to day open when we want them and close when we're done, like a browser or a photo editor. A daemon isn't that. It's a program meant to stay running, with no screen and no button to press, quietly repeating its work in the background. The word comes from the Unix world, pronounced "dee-mon," meaning a helper that works away behind the scenes without ever showing itself.

With no screen, it needs something to look after it instead of a person. On Linux that something is systemd or cron. Their job is to start the daemon when the machine boots and wake it back up if it falls over. Sounds safe enough, except, as you'll see, that very waker is sometimes the thing that fools you.

What ours actually do

In our house (the small fleet of systems we run ourselves) there are several daemons, each doing a different job:

• one that fetches product prices from online shops and updates them every morning
• one that listens to chat messages and answers or routes them to the right place
• one that summarizes the data flowing in all day and keeps it as notes to read back
• one that watches spending and warns if anything runs over the budget we set

The charm of a daemon is that you set it once and get to forget it, no pressing buttons every day. But that same charm hides the trap. A thing you get to forget is also a thing nobody is looking at. The day it goes quiet, you don't notice, because its silence on a normal day and its silence on a dead day look exactly alike. Throughout this article we'll call it "dying" when a daemon stops working, the word programmers actually use.

Part 2Why daemons die so silently

The daemons that died quietly in our house all had the same shape. Something inside falls over, but a soft fallback catches it, so the surface keeps working, nobody knows, and it drags on. Here are four I hit myself.

The one that died for two weeks

One daemon acted as a bridge to another machine. The day we moved that machine, its address changed, but the code still had the old address baked in. So on start it fell over within four seconds. Here's the thing: we'd told systemd "if it falls, always wake it back up." The waker did its job, woke it, it fell, woke it, it fell, until it gave up and quietly stopped trying. The very thing we set to keep it alive became the thing that covered the body over. Two weeks passed before we knew it was dead.

The brain quietly swapped for a smaller one

Another group of daemons need to call a capable AI as their brain. One day the key that proves their identity expired. Unable to reach the capable one, they fell back to a small model running locally instead. The result: every one of them still answered, just answered dumber. Because they still answered, nobody thought to ask, and it ran four days before we caught it. The only signal that surfaced was a small error in a single job.

The one that never worked from day one

This one is worse. A script was written to give a status summary every time we start work. It turned out to have an error on its very first line, but the code wrapped it in a "if this breaks, never mind." So the moment it broke, it just calmly answered "no data" every time. We read "no data" and assumed it was working but finding nothing. In truth it never once ran successfully in its whole life. It just failed so politely it looked normal.

The money-watcher that fell without a trace

The last one watched spending. It died, and the scheduler (cron) kept trying to call it several times a day, called it and it crashed, over two hundred times in total, but not a single log line was written. It failed so cleanly that going back to look, there was no trace to find.

The shared lesson across all four

Look across the four and the same core shows up. Every one had a good fallback that kept the surface looking fine, and that is exactly the problem. The softer the fallback, the more silent the death. The gentleness we deliberately built in so the system wouldn't crash was, at the same time, erasing the warning. Dying loudly would have been better, because at least we'd have heard it.

Part 3Make it make a sound when it goes

The fix isn't making a daemon that can never fail, because everything fails eventually. Someday the key expires, the address changes, the machine restarts. The real fix is making sure that the day it fails, you hear something. Three things turn the silence into a sound.

1. A fallback must always announce itself

Every time a system drops to its fallback, don't let it drop quietly. Force it to emit at least one signal: write a log line, or raise a flag in a status file, saying "I'm on the backup right now." Go back to the brain swapped for a smaller one. If, the moment it dropped to the small model, it had written one line that it was on the fallback, we'd have caught it in minutes, not four days. Dropping to a backup without telling anyone is a time bomb you set yourself.

2. An alert must prove it was delivered, and carry a heartbeat

An alert that fires and then assumes it arrived is an alert you can't trust. We once had a watcher that fired an alert but never checked whether the other end received it, then marked itself "already alerted." So on the day the message didn't actually get through, it remembered it had alerted and never fired again. The undelivered alert was lost for good. A good alert has to confirm the other end really received it, and it has to send a heartbeat on a rhythm, so you can tell "quiet because everything's fine" apart from "quiet because the alarm itself is dead." They are not the same thing.

3. Check deeper than "is it still answering"

The lesson from the smaller brain is that asking "is the daemon still answering" isn't enough, because it did answer, just with something degraded. The question to ask is "answering with which brain?" Checking that it's alive has to reach down to whether it's still running on the right thing, not just seeing a reply come back and feeling reassured.

The three layers we put around this

Once the lessons added up, we set watchers in three stacked layers. The idea is plain:

1. Bottom layer something that actually exercises each daemon every day, not just checking that it's up, but sending it real work and seeing if the right thing comes back.
2. Middle layer the bottom layer's results roll up into a status summary a human reads regularly: which are green, which are red, which have gone unusually quiet.
3. Top layer a surface you see all the time, like a status light that stays on. When something isn't okay, it turns a warning color that catches the eye.

The key to the three is that the top layer watches the bottom too, including checking that the signal from below is still "fresh," not left over from yesterday, because otherwise the watcher itself can die silently just the same. How to wire the three layers so they run on their own is the part we put the work into and keep to ourselves, but the three principles above you can use today with no special tooling.

The short version before we close

A daemon is a program that runs in the background all the time. The upside is you can set it and forget it. The downside is the very same thing: once you can forget it, the day it dies goes unnoticed. And the better the fallback you add, the more silently it dies. The one rule left is this: anything that runs quietly on its own has to be forced to make a sound the moment it starts going wrong.

Look around you. At home or at work there's almost always something running quietly on its own: the backup set to run at midnight, the script that sends a report every morning, the plugin syncing your files. The question worth asking isn't "will it fail," because someday it will. It's "the day it fails, will I hear anything at all?" If the answer is dead silence, that's the place to start.