My LLM Summary Bot Said 'No Intervention Needed' for Three Weeks Straight

Paper trading. Returns and ratios are real ledger values; the budget is the real configured value.

I bolted an LLM commentary panel onto a bot dashboard. Each day it serializes state to JSON, hands it to a local CLI, and gets back a five-line summary. The last line is always a verdict.

For three weeks, that verdict read:

Summary: ledger consistent, running to plan, no intervention needed.

What actually happened over those three weeks: against simply putting the same budget into the same ticker and doing nothing, the bot was -20.38pp. The sibling bot on another ticker: -35.47pp.

The model did not summarize badly. Given what I handed it, no other sentence was available.

Is your AI summary panel receiving input that lets it deliver bad news?

What I gave the model

The state JSON looked like this:

{
  "symbol": "...",
  "cycle_no": 2,
  "shares": 3,
  "avg_price_usd": 73.23,
  "invested_usd": 219.69,
  "carryover_usd": 122.78,
  "realized_pnl_usd": 16.61,
  "commission_usd": 0.0,
  "cumulative_loss_usd": 0.0,
  "ledger_ok": true
}

Line by line these are all reasonable operational fields. But sort them by what they actually measure:

• Rule-compliance fields: cycle_no, shares, invested_usd, ledger_ok, cumulative_loss_usd
• Performance fields: realized_pnl_usd, and that's it
• Comparison fields: none

One performance field, and it read +16.61. Positive. The loss-limit field read 0. No violations.

A model receiving this input has structurally one conclusion available: "no rule violations + positive realized P&L = going well."

Feed only absolute values and you get optimism. Not because the model is optimistic, but because you gave it no grounds for pessimism.

This strategy is supposed to lose to the benchmark

Here is the important twist. This bot underperforming buy-and-hold is not a bug.

An averaging-down strategy splits the budget into 40 tranches. At the three-week mark, 6% of the budget was deployed and 94% was sitting in cash. Cash earns nothing in a rising market. Losing is the expected outcome.

More damning: I already knew this. The backtest document written before deployment contains this section:

The most important finding: expected return is far below buy-and-hold

Across full history, the strategy's +499% realized return is less than 1.7% of buy-and-hold's +29,214% over the same period.

I knew it, wrote it down, committed it. And never put it on the operational screen.

That is the real failure. Knowing something and having it on screen are different things. What I look at every morning is not the document — it is the dashboard, and the dashboard said "+$16.61" and "no intervention needed."

Would you fix the prompt

There is a fork here.

1. Fix the prompt — add "don't be optimistic," "be critical" to the system prompt
2. Fix the input — put a comparison into the JSON

Option 1 is tempting. It is one line. And the tone does change — the model starts hedging.

But if the data is unchanged, the model is just hedging about facts it doesn't have. You get sentences like "realized P&L is positive, though a cautious approach is warranted." Zero information content, and it looks balanced, which makes it worse.

I took option 2. Prompt tuning cannot fill a data hole.

Making it apples to apples

The comparison math had its own trap. My first version was:

strategy return = realized P&L / budget

That makes the strategy look worse than it is, because unrealized gains on the open position are missing. The benchmark reflects all mark-to-market movement; measuring only the strategy's realized portion isn't a comparison.

The corrected definition:

unrealized  = shares * last_price - cost_basis
strat_value = budget + realized_pnl + unrealized
strat_ret   = strat_value / budget - 1
hold_ret    = last_price / first_fill_price - 1

Three anchors have to match: same budget, same start point (the ledger's first fill date and price), same valuation basis (unrealized included). Break any one of them and the number you produce cannot be used as evidence.

I also exposed cash_idle_pct. That field is what makes the cause of the underperformance legible — not a bad signal, just undeployed capital.

The changed output

I added a benchmark_vs_buyhold field to the JSON and one line to the system prompt:

If benchmark_vs_buyhold is present, you must devote one line to excess return versus same-budget buy-and-hold. Never write "going well" on realized P&L alone. State explicitly that underperformance in a rising market is by design, not failure.

Same bot, same day, new output:

Excess return versus same-budget buy-and-hold: -20.38pp (strategy 0.59% vs hold 20.97%). 93.6% of the $3,424 budget sits in cash; underperformance in a rising market follows from this design and is not a failure. The judgment criterion is drawdown defense, which remains unverified as no decline has occurred. Realized P&L is $16.61, but that figure alone cannot establish performance. Cumulative loss $0, ledger consistent — running to plan, no intervention needed.

The last line is unchanged. And it is correct. The bot is running to plan, and there is no reason to intervene today. What changed is that the sentence now sits on top of four lines of context.

Even at the same conclusion, "no intervention needed" derived from a comparison is a different sentence than one derived from absolute values alone.

Three-line self-check

If you delegate summaries, verdicts or reports to an LLM:

1. How many fields in your input JSON are comparative? If zero, that summary is structurally biased toward optimism, regardless of tone.
2. Which field would the model have to read to say "this is bad"? If it isn't in the input, the model cannot say it.
3. Are the limitations you wrote in your docs also on the operational screen? Knowing and looking-at-daily are different. Looking-at-daily wins.

The honest part

This bot still loses to the benchmark. It's still -20.38pp after the fix. That doesn't change — it's a strategy that keeps 94% of its budget in cash.

What changed is that the fact is on the screen. And that is what makes judgment possible. This strategy's claim was never "earns more," it was "smaller drawdown for the same budget," and that only gets tested when a decline arrives. Until then it is undecided, and there is no switch to a live account.

One more thing. The reason this went unnoticed for three weeks is that the commentary never said anything false. "Ledger consistent" was true. "Cumulative loss $0" was true. It was a wrong impression assembled entirely from true statements. Same family as the monitor that lied a different way and the collection cap that wrote my conclusion for me — except this time there wasn't even a bug in the code. One missing field was the whole story.

If you have an AI summarizing state right now, open that input JSON and read the fields one at a time. How many of them answer "compared to what?"