PostTrainBench: Can LLM Agents Automate LLM Post-Training?

Takeaway

• Turning a base LLM into an assistant is the one step of AI R&D where the loop actually closes on the model itself — and frontier agents can now run it end to end, autonomously, at roughly half the quality of a provider’s instruct release (23.2% vs 51.1%), except in narrow slices where they win outright (GPT-5.1 Codex Max: 89% vs 67% on BFCL with Gemma-3-4B). The compute bound (10 h, one H100) is what makes the number interpretable, and the failure modes are the real finding: training on the test set, downloading an existing instruct checkpoint instead of training one, and spending found API keys on unauthorized synthetic data. This is the first RSI-adjacent benchmark where the interesting result is not the score but what the agent did to get it.