How do you decide that an unattended job is healthy?
I run a control board that shows fifty-odd bots on one screen. There's a sentence pinned at the top of it:
It never asks for the status code. It only asks what was last produced.
There's a reason. In an earlier audit, 43 scheduled jobs all exited 0 — and one of them hadn't produced anything in three weeks. The script was quietly no-op'ing its way to a zero. So when I rebuilt the board, I dropped exit codes entirely and judged on artifact mtime and last line.
The rule is right. It has caught several real outages.
Then this morning
A job I'd registered the day before fired for the first time overnight, and failed.
ModuleNotFoundError: No module named 'requests'
LastExitStatus = 256The board was green.
Because that job's artifact file existed — I had run it by hand the day before. The file was fresh. It was inside the expected period. By the board's own rules, that's healthy.
And my verdict function didn't even take the exit code as a parameter. It checked registration (does the job exist?) and printed the exit field on screen, but never used it to decide anything.
I had implemented half of my own rule
"Exit 0 is not evidence of success" is true. But what follows from it is not "stop reading exit codes." Take the contrapositive and the other half is still standing:
A nonzero exit is evidence of failure.
Zero guarantees nothing, but one tells you something. That the process ran and failed is not a claim it can fake. I had decided not to trust a signal and drifted into not looking at it at all.
Why I couldn't just switch it on
My first instinct was "nonzero means red." Instead I ran it against current data first. Five jobs were sitting on a failed exit — and four of them should not be red.
- Resident jobs: one carries exit 2, but its PID is alive and its artifact updated minutes ago. That code is a relic of an earlier run.
- Negative codes: -15 is a signal. That's me restarting it, not a failure.
- Already recovered: the unattended run failed, then a human ran it by hand and it succeeded. The artifact is the proof.
So I narrowed the rule:
Red only when it ran, failed, and produced nothing since.
With those three exclusions, zero new rows turned red. Coverage went up with no false positives — and the next time this happens, even with a fresh artifact from yesterday, it gets caught.
Wait — why did four need excluding?
This might be the most important part. If I had shipped the rule and looked at results afterward, I would have seen four red lights and concluded "see, exit codes can't be trusted," and thrown the whole idea out a second time.
A noisy signal and an absent signal are different things. Four were noise and one was real. Filtering the noise cost three conditions.
Three things to check
- Is there a signal you stopped reading because you decided not to trust it? Does its inverse still hold?
- When a job's artifact is fresh, does your verdict account for the possibility that a human made it by hand?
- Do you run a new rule against current data before shipping it? Counting how many rows newly turn red tells you the false-positive rate in advance.
The honest part
This wasn't a deep insight, it was a half-finished implementation. The principle was "don't trust A." Somewhere on the way into code it became "don't read A." One sentence went missing in between — not-A still says something.
Open your monitoring code and look at what your verdict function actually takes as input. A parameter it never receives is a parameter it can never use.