Our response to Epoch AI's review of PostTrainBench
Epoch AI designated PostTrainBench as Verified. We are preparing version 1.2 to address the issues flagged in the review and additional issues that we identified.
Epoch AI designated PostTrainBench as Verified.
We found the report to be very detailed and accurate, and we appreciated the depth of its analysis and attention to detail. We also identified some of these shortcomings independently. We will soon release version 1.2, which addresses most of the major issues flagged in the report, as well as some additional issues that we identified.
The responses below follow the sections in Epoch AI’s review. For each point, we quote the relevant finding and respond directly.
Our response
“Interpretation”
Narrowly fitting a single eval
Epoch AI's review“Models can achieve high scores by narrowly (over)fitting to the single eval rather than teaching general, transferable skills to the base model.”
This is the intended behaviour. Each agent gets one H100 for ten hours to post-train a base model. Within that budget, teaching broad, transferable skills is not realistic, so we ask a narrower question: can the agent improve the model on one target benchmark?
A middling average and judge variance
Epoch AI's review“A middling average can mix strong legitimate runs with disqualified ones. High variance can be created either by inconsistent models or a mixture of strong runs and runs flagged as contaminated.”
We manually verify most runs flagged by the judge to make sure the judge is calibrated and the reason for flagging matches the agent's behaviour.
That being said, we agree that high variance can be caused from the agent side through inconsistency, or disparate run results through flagged and unflagged runs. To further account for judge variance and improve reliability, we are planning on a best of three judge verdict for PostTrainBench v1.2.
Uneven gains across benchmarks
Epoch AI's review“Gains are generally uneven across benchmarks. An average score can reflect strong performance on easier tasks and limited progress on the hardest ones rather than moderate improvement across the board.”
We designed PostTrainBench to measure progress towards automated post-training at varying levels of difficulty. Therefore, we use several different, increasingly difficult benchmarks as targets. More difficult benchmarks are given greater weighting in order to accurately reflect gains, as described in Table 5 of our paper.
Our response
“Task analysis”
BFCL
Epoch AI's review“We classify one of the included benchmarks, BFCL, as Flawed due to an observed error rate of 48%, but the selected subset used in PostTrainBench may have a different error profile.”
We agree with this analysis of BFCL and are planning to remove it from PostTrainBench v1.2.
Trace coverage
Epoch AI's review“We analyzed the traces produced by Opus 5, the only agent with a complete set of v1.1 traces.”
All agents have the full set of traces. For most agents, the traces are a mixture of v1.1 and v1.0. This is intentional, as we only ran v1.1 on flagged runs for older agents because the unflagged runs did not need to be rerun under the new settings.
Opus 5 is the only agent which was released after the v1.1 ruleset and thus has a full set of v1.1 traces. Other agents have a mixture, where we reran on the v1.1 setting if it was needed. This is highlighted in the trace viewer when present.
Public viewer
Epoch AI's review“The public viewer only shows traces and verdicts for contamination and model identity, not API-distillation and PostTrainBench lookup.”
This was an oversight. We have now updated the public viewer to show the API and PostTrainBench lookup verdicts too.
Number of runs
Epoch AI's review“The original paper mentions three runs, but we limit our review on v1.1, which ran just twice.”
The majority of agents ran for three runs. For more expensive agents we had to run twice due to our compute constraints:
- Fable 5
- GPT-5.6 Sol
- Opus 5
- Opus 4.8 High
- Opus 4.8 Max
- GPT-5.5 xHigh
- Grok 4.5
- GPT-5.1 Codex Max High
- Kimi K3
- GLM 5.2
- Opus 4.7
- Gemini 3.1 Pro
- GPT-5.2
- GPT-5.1 Codex Max
- GPT-5.4 High
- Gemini 3 Pro
- Opus 4.6 1M
- GPT-5.3 Codex High
- GPT-5.2 Codex
- Opus 4.6
- Opus 4.5
- GPT-5.3 Codex Medium
Our response
“Limitations”
Distillation and previous results
Epoch AI's review“Distillation and reuse of previous results are legitimate, widely used post-training techniques, but are banned here.”
Distillation is permitted. Agents can distill from locally served models. However, it is disallowed to distill from a model served through an API. Such distillation could violate the benchmark constraint of one H100.
We do not allow looking up PostTrainBench traces or previous runs to ensure independent, comparable results. Part of the purpose of PostTrainBench is to measure how well agents can conduct literature reviews and exploratory experiments to converge on a good strategy.
LLM judges
Epoch AI's review“LLM judges are used to check for cheating behaviors. These have documented biases and can be unreliable, especially when grading non-discrete behaviors.”
We agree that unreliability is a concern regarding LLM judges. We address this concern in two ways. First, we manually review some of the traces to ensure the verdicts are reasonable and calibrated. When necessary, we update the judge to a newer model or update the prompt and overall design of the judges. These were our main updates in version 1.1.
When updating to v1.1, we tested various models from different providers and selected the most effective one. Second, in version 1.2, we plan to run the same judge three times and use a majority vote to determine the final verdict to further reduce noise.
Judge traces
Epoch AI's review“Judge traces for the other two checks, API-distillation and PostTrainBench lookup, aren't provided, and thus were not evaluated.”
We have now updated the trace viewer to also show the API and PostTrainBench lookup checks.