03Agentic AI · Energy markets
European power research agent
Can an LLM produce better multi-step power-market research when it retains its reasoning and accumulated evidence rather than repeatedly reconstructing the investigation?
Inspired by OpenAI’s ARC-AGI-3 harness research, I built and preregistered a controlled OpenAI Responses API experiment using GPT-5.6 Sol at max reasoning. Eight hash-locked synthetic European power-market investigations each required five sequential evidence gates. Every case was run twice across three conditions: stateless with a three-action rolling history, retained reasoning, and retained reasoning with deliberately forced compaction—48 randomized API runs in total.
Retained reasoning achieved 16/16 exact successes and a 100/100 mean score, versus 0/16 and 0/100 for the stateless baseline. The paired episode-level difference was +100 points (95% CI [100, 100], exact sign-flip p=0.0078), while using 35.2% fewer output tokens and costing 41.9% less. Every stateless run exhausted the fixed step budget after losing and reconstructing earlier evidence.
The primary contrast saturated: all eight episode-level score differences were +100. That is why the bootstrap interval collapsed to [100, 100], while p=0.0078 is the minimum attainable two-sided exact sign-flip value with eight non-zero episode differences. The result supports retention in this defined stress test but does not provide fine-grained effect-size resolution.
Forced compaction activated in all 16 designated runs, but it did not meet the preregistered quality-preservation or output-token-reduction rules. It scored 97.5/100 with 14/16 exact successes, used 8.0% more output tokens, cost 39.2% more and took 198.6% longer than retained reasoning alone.
PythonGPT-5.6 SolResponses APIPreregistered evaluation
Decision: The experiment supports retained reasoning for this defined synthetic memory-stress workflow. It does not support a compaction benefit at the aggressive 1,000-token threshold, and it does not claim live-market alpha or universal generalisation.
Read the evidence +
What I built
I preregistered eight hash-locked synthetic investigations with five sequential evidence gates, then ran every case twice across three randomized harness conditions using the same GPT-5.6 Sol model, max reasoning, tools, evaluator and fixed eight-step budget: 48 API runs.
Validation
The paired analysis treated the eight episodes—not the repeated draws—as the independent units. All eight episode-level score differences were +100, so the episode-clustered bootstrap interval collapsed to [100, 100]. The exact two-sided sign-flip p-value of 0.0078 is the minimum attainable with eight non-zero episode differences.
What I found
Retained reasoning scored 100/100 with 16/16 exact successes, compared with 0/100 and 0/16 for the stateless baseline, while using 35.2% fewer output tokens and costing 41.9% less. All stateless runs exhausted the step budget after reconstructing discarded evidence.
Boundary
This deliberate memory-stress benchmark saturated on the primary contrast, so it supports retention for this defined synthetic workflow but cannot finely resolve the effect size or establish generalisation to live research. Forced compaction scored 97.5/100 with 14/16 exact successes, used 8.0% more output tokens, cost 39.2% more and took 198.6% longer than retention alone, and did not meet the preregistered quality-preservation or token-reduction rules.
