The Token Had Eight Hours Left When It Said 'Session Expired'

Two unattended jobs died with the same line:

Failed to authenticate: OAuth session expired and could not be refreshed

It reads like an instruction. The session expired, so log in again. I did exactly that, for three weeks.

Does your error message name the cause, or just the symptom?

The re-login didn't survive the same evening

On September 16 at 09:30 I logged in by hand and verified it worked. It did.

At 16:00 the same day, the weekly documentary job died on that line. At 18:05, the job that turns blog posts into video scripts died on the same line.

So much for "re-login lasts a few days." The expiry hypothesis was already wobbling — but I still had nothing that could falsify it, because nobody recorded what the file looked like at the moment of failure.

So I photographed the moment of failure

Instead of fixing anything, I added instrumentation. On every failed CLI call, log the credential state immediately before and after — no token values, only an 8-character SHA-1 fingerprint and the remaining lifetime.

18:05:06  claude_fail  rc=1 OAuth session expired and could not be refreshed
18:05:06  cred  before[mtime_age=90s exp_in=28708s at=d3b42fa6 procs=6]

exp_in=28708. That is 7 hours 58 minutes of remaining life.

The job did not die holding an expired token. It died holding a perfectly valid one. For three weeks I had been repairing something that was never broken.

Then who broke it?

A separate probe samples the credential file every two minutes. Here is that evening:

Two rows stand out.

17:41: the freshly written token has a negative lifetime. A value that had expired a day and a half earlier was written into the file at that moment.

17:43, 18:01, 18:03: the fingerprint is da39a3ee — the first eight characters of the SHA-1 of the empty string. The file existed, and the token fields were empty.

What would you suspect here?

Corrupt storage? A bad writer? I looked at the process list.

4918   Mon Sep 14 09:04   02-23:48   (interactive session)
26217  Tue Sep 15 09:12   01-23:40   (interactive session)
68713  Tue Sep 15 10:08   01-22:45   (interactive session, stuck in a wait loop)

Only a process that has been alive for more than a day and a half can be holding a day-and-a-half-old token in memory. Four interactive sessions, left open in terminal tabs and forgotten. One had been sleeping in a wait loop for two days.

A long-lived session writes its stale in-memory credential back to the file, and whichever unattended job happens to authenticate in that window dies.

That also explains "it only fails at unattended times." Not because unattended environments are special, but because that is when several human sessions happen to be alive. Two weeks ago I retired the contention hypothesis because no two scheduled jobs overlapped on the same minute. What overlapped wasn't the jobs. It was the terminals.

Bonus: my counting tool was hiding the culprit

I was counting concurrent processes with pgrep -fl claude. At one instant it reported 5. With -af, the same instant reported 7.

pgrep omits the calling process's own ancestors — which is exactly the oldest session in the room. The tool counting the suspects was leaving the suspect off the list.

Three things to check in your own setup

1. Does your failure log record the state at the moment of failure, or only that a failure happened?
2. How many processes write to your shared credential file? Are any of them long-lived?
3. Does your process-counting code count itself and its parents?

The honest part

This is not a confirmed root cause yet. If write-back is the mechanism, clearing the old sessions should stop the recurrence. So today's work was not a fix. It was two things: clearing sessions older than 24 hours, and making the probe dump the full process list into the ledger the next time an empty or already-expired token gets written.

For three weeks I read an error message and did what the sentence told me to do. The sentence was a symptom, and a symptom is not a cause. The bug that lived eight extra days because the gate only caught zero is the same family — if you never suspect the instrument, you keep changing the wrong side.

Pick one unattended job of yours and check whether its last failure line has the state of the world printed next to it. If not, the next failure ends in guesswork too.