JM.Back to work ↗
Applied AI · Power research agent

Does giving an AI room to reason make it better?

On this defined test, retained reasoning did. The result is bounded to the synthetic suite.

16/16retained-reasoning successes
Decision

Retained reasoning won this defined synthetic memory-stress test; compaction did not pass its rules.

Controlled memory-stress test48 runs · synthetic suite
Retaining the reasoning state passed every defined episode. Compaction was cheaper than stateless, but it did not pass its preregistered rules.
Why it mattered

When an agent forgets its own investigation

An agent can waste its limited steps rediscovering evidence it has already seen. I designed a test that made this failure visible while holding the questions and scoring rules fixed.

What I built

Three memory conditions

I ran the same eight synthetic investigations twice under each memory setting: a short rolling history, retained reasoning, and retained reasoning with compression. Tools performed the calculations; the model chose which evidence to inspect.

What the logs showed

Forgetting consumed the step budget

Every stateless run used all eight allowed steps, repeatedly reconstructing lost evidence. Retained reasoning completed all sixteen runs successfully. This suite was deliberately designed to stress memory loss.

What I would test next

A broader test of the same idea

A larger, independently written hidden set would test whether the result extends beyond these cases. Compression also needs testing at realistic long-context sizes before making a production recommendation.

Technical evidence

The harness ran 48 controlled cases across eight hash-locked investigations. Retained reasoning passed 16/16 exact tests; stateless history passed 0/16; reasoning plus compaction passed 14/16 but failed its rules. The episode-clustered 95% interval for the primary score difference was [100, 100]. The exact-test p = 0.0078 is the floor for eight independent episodes, not a claim of universal improvement or live-market alpha. The retained condition also cost 41.9% less than the stateless run.