My alert code was only failing inside the scheduler

The code that sends alerts couldn't send alerts

When you run several bots alone, what you want is simple: your phone buzzes when something changes or breaks. So I wrote a send() function that posts to Telegram whenever a gate changes state. It was a few lines of Python using the standard library's urllib to call the Bot API.

On this machine, that request failed at the SSL step. The Mac has a corporate self-signed certificate chain installed, and urllib couldn't get through it. The system curl on the same machine went through fine.

The second trap was launchd. Run from a terminal, the token is in the shell's environment and everything works. Run as a launchd job, that environment isn't injected. It passed when I tested it in a shell, but when it ran on schedule the credentials weren't there.

If you've ever run your alert code once from a terminal and decided "it works," did that check use the same network path and the same environment your scheduler uses?

Same function, different world

What confused me is that the code itself had no bug. The URL was right and so was the payload. Two conditions of the run were different:

• The trust path for TLS verification: the Python runtime and system curl don't use the same one, and this machine's certificate chain only broke the Python side.
• The environment: variables that exist in an interactive shell don't exist under launchd.

So what I saw while developing and what the daily job actually did came out different.

What would you do? Dig into the runtime's certificate setup and fix the standard-library path, or borrow the path that already works on this machine?

The fix: system curl and a 0600 file

I took the second option.

• send() now shells out to system curl and uses the TLS path that already works on this machine.
• Credentials come from the environment first. If they're missing, they're read from a fleet_alert.env file in a config folder under my home directory, with permissions set to 0600. That way the launchd job gets the same values without anything injected into its environment.

Pointing the runtime at a certificate bundle would probably also have worked. This commit went with curl, which means the fix is tied to this machine.

The other half: what counts as broken?

Once alerts could get out, I had to decide what to send. Apart from gate transitions, I wanted to know when a bot broke. So I added fleet_events.py:

• It scans the autopilot logs of five bots, including shorts, Threads and Instagram.
• It only picks lines with breakage markers: Traceback, upload/publish failures, token expiry, quota errors and HTTP 4xx/5xx.
• It skips intentional no-ops, such as lines for insufficient signal, dry runs and waiting on a gate.

The third rule mattered more than I expected. My bots skip work on purpose a lot. They rest when the signal is weak, don't publish on a dry run and wait when a gate is closed. Those lines look like "nothing happened," but they mean the bot did what it was designed to do. If they triggered alerts, the real breakage alerts would get lost among them.

To avoid resending the same line every day, the scan works by byte offset. It saves how far it has read in each file and only looks at lines added after that point, so each line is alerted once. If a log gets truncated below the saved offset, the scan starts over from the beginning. A missing file is skipped.

Both scripts are wired into the daily run of the shorts autopilot, which scans every log once a day.

There are four tests: markers are caught and no-ops are skipped, the offset moves forward, truncation resets it, and a missing file doesn't crash anything.

Self-check

• Have you sent at least one alert from the same environment your scheduler (launchd, cron) uses, rather than from a terminal?
• Does your log watcher keep an explicit list that tells intentional no-op lines apart from breakage lines?
• If you read logs by offset, have you tested what happens when a file is truncated or goes missing?

The honest part

There's no record of how many alerts failed to go out before this commit, so I don't know. Breakage detection matches string markers, so any failure message in a format that isn't on the list gets missed. The scan runs once a day, so this isn't real-time monitoring. The curl workaround depends on this machine's certificate setup and will need checking again on any other machine. And a 0600 config file is still a plaintext file.