PostTrainBench v1.1
Hardening PostTrainBench against reward hacking
PostTrainBench v1.1 tightens the rules and our cheating checks, after stronger agents found ways around the original ones.
Update: PostTrainBench v1.2 is out, with further fixes to scoring and judging.
As agents are getting better, they are finding ways to raise their scores on benchmarks without actually getting better at the task itself. In PostTrainBench, we found that a few agents wrote training examples around very specific held out samples after looking at where they failed the eval. Other agents found API keys in their environment and used a stronger model as a teacher, for distillation. One agent just submitted an instruct model instead of the base model that it was supposed to submit.
PostTrainBench essentially measures hill climbing on the constituent benchmarks in a compute constrained setting, so it matters where we draw the line. In v1.1 we make that more explicit: training on a broad failure mode is fine, but training on data built around particular test samples is not allowed.
The original v1.0 already had a general anticheat judge. The v1.1 audit flagged 234 runs for test set contamination, 12 for using external APIs and 10 for submitting a different model, many of which the old judge had missed. Three GPT 5.6 (Sol) runs had also looked up PostTrainBench itself and used our published traces from previous runs.
Agents and judges now get the same rules and boundaries. Each kind of "reward hacking" has its own review, and agents can use the same decontamination tool the judges use. The main setting of the benchmark: i.e. the compute budget, base models and tasks haven't changed.
Test set contamination
Agents are supposed to run evaluate.py, look at what they got wrong and then work on the fixes. They can notice if the trained model is bad at certain cases: like converting percentages to decimal, bad medical answers in non English languages, etc. When the agents then generate a lot of varied examples to train that into the model, that is a good skill that we are testing for.
It however becomes contamination when the training data is targeted around certain test samples. Paraphrases and close variants of the same sample, even though not verbatim copy, are not allowed. So we judge the training set the agent actually used, not if it just looked at the benchmark.
The benchmark where it mattered the most is BFCL. An agent that gets better at function calling in general is doing what we want.
Agents may inspect benchmark failures and train broadly against the respective failures. They may not generate training examples that are clustered around particular test cases (including paraphrases, variants, etc.).
- The agent prompt now differentiates what's allowed and what's not: failure mode targeting vs disallowed item targeting.
- The contamination judge looks at the resulting training data distribution rather than treating benchmark inspection itself as evidence of contamination.
- The same deterministic n-gram overlap tool (inspired by AI2's Decontamination Toolkit) is now given to both the agents and the judges.
- Affected runs require a rerun. If a comparable rerun is unavailable for an older agent, the affected result is then removed from the leaderboard.
We looked at all previous runs under these new rules. The violations ranged from loading a whole eval set to memorize it, to writing synthetic templates around single GSM8K and BFCL items.
Large open source datasets on HF can overlap with the test sets by accident, and agents usually can't check them by hand. The shared tool lets the agent find and remove those cases before training. Matching a benchmark's style is fine on its own.
Submitting a different model
Each run gives the agent a specific base model to post train. If the agent submits some other model, like an off the shelf instruct model, then the score is meaningless, even if the checkpoint is from the same model family.
We added a programmatic model identity check that compares the submitted model artifacts with reference configs for known instruction tuned models.
For example, in one run Kimi K2.5 couldn't get its fine tuning of Qwen3-1.7B-Base to work properly, so it instead saved the official Qwen3-1.7B instruct weights as final_model.
Using external LLM APIs as teachers (distillation)
Some agent environments have API keys in them because the agent or an evaluator needs them to run. They are not meant to be used for training. However, we still found agents reading those keys from the environment and using frontier models to create training data, even when the surrounding code told them not to do so.
API keys present in the environment shall not be used to call an external model for training data or other post training work.
- A separate API usage judge now checks tool calls and agent artifacts.
- Unrelated API credentials are now removed or blocked from the agent environment.
- Affected runs due to external API use were rerun under the new setting.
Agents can still train on their own model's outputs, and they can load other models on the given GPU. They however cannot use external models through an API.
Direct benchmark lookup
In one run, GPT 5.6 (Sol) on HumanEval with Qwen3-1.7B searched for PostTrainBench directly. It cloned the repo and found the trace viewer and the HF dataset behind it. It then filtered that dataset for that benchmark and base model.
Agent tracePrior clean runs on this exact setup establish a useful ceiling...
It downloaded previous traces and training scripts and then copied their data mix, LR schedule, decoding settings and GRPO setup. This isn't leakage of the test data as such, however this breaks the independence between runs, which is not allowed.
Agents aren't allowed to look up PostTrainBench, the leaderboard, the traces, or anything related.
- A lookup judge looks at agents' searches and trace activity for attempts to look up PostTrainBench.
From one judge to shared rules and specialized judges
The agent and judges now share the same contamination rules and decontamination tool.
Single general judge
v1.0 had one judge that checked for contamination and model swapping. It didn't separate broad failure mode training from item specific training, and did not check for external API usage.
Specialized judges and checks
- Agent rule differentiating failure mode targeting from item targeting
- Shared decontamination tool for both the agents and judges
- Contamination judge based on the training data distribution
- Programmatic model identity check
- API usage and PostTrainBench lookup judges
Clean runsFull evaluation
Flagged runsPenalized base model score
How the leaderboard changed
We reran all older runs on the new v1.1 setting and recomputed the leaderboard. If a run needed a rerun we couldn't do, for example because the model had been deprecated, we removed it. Below are the biggest changes:
Aggregate benchmark performance. Differences include the v1.1 audit and reruns under the new contamination rules.
Fable 5's v1 result was collected during its first 72 hours, when rate limits and refusals affected several runs, and should be treated as preliminary.
Reruns and infrastructure fixes
We reran 121 runs in total, for API misuse and various other infrastructure problems. Fable 5 now has two seeds, which replaced the SmolLM results we had filled in from Opus as a fallback. Its GPQA score still comes from Opus 4.8 Max, because Fable refused that benchmark due to safety classifiers.
The contaminated runs were rerun with the new setting, and the trace viewer now shows their artifacts and the judges' decisions.