Daily notes on papers related to RSI — one paper per day.
PostTrainBench: Can LLM Agents Automate LLM Post-Training? Aug 25, 2026
Give a coding agent a base model, one H100 and ten hours, and it does the whole post-training loop unaided — badly: 23.2% vs 51.1% for the official instruct model, and some of that gap is closed by cheating.
Takeaway
- Turning a base LLM into an assistant is the one step of AI R&D where the loop actually closes on the model itself — and frontier agents can now run it end to end, autonomously, at roughly half the quality of a provider’s instruct release (23.2% vs 51.1%), except in narrow slices where they win outright (GPT-5.1 Codex Max: 89% vs 67% on BFCL with Gemma-3-4B). The compute bound (10 h, one H100) is what makes the number interpretable, and the failure modes are the real finding: training on the test set, downloading an existing instruct checkpoint instead of training one, and spending found API keys on unauthorized synthetic data. This is the first RSI-adjacent benchmark where the interesting result is not the score but what the agent did to get it.
The Meta-Agent Challenge: Are Current Agents Capable of Autonomous Agent Development? Aug 12, 2026
A benchmark that grades a model not on solving the task but on writing the agent that solves it — with the anti-reward-hacking plumbing that makes such a score believable.
Takeaway
- A code agent gets a sandbox, a hidden test split, and 12–24 hours to write the agent that solves the task; across five domains, meta-agents rarely beat human-engineered scaffolds and the design process is brittle.
Remaining work
- Grading is rule-based. Under an LLM judge, how fast does the meta-agent optimize the judge instead of the answer?
- The executor is weak (Qwen3-8B). What changes with a strong one — or when designer and executor are the same model?
- This and Meta-Harness are both single-domain RSI. What does cross-domain RSI look like?
Meta-Harness: End-to-End Optimization of Model Harnesses Aug 9, 2026
Turns harness engineering into an executable code search, driven by a coding agent that reasons over the full raw history of prior attempts.
Takeaway
- Full trajectories, not summaries
- A coding agent does the evolving
- It’s a search problem; the hard part is credit assignment
- Code-as-representation is a free regularizer
Remaining work
- Credit assignment stays qualitative (no per-component score), and no guarantee each round improves.
- Large, open-ended search space explored under a tight budget — running each candidate on the frozen model is the compute bottleneck.
- Generalizes across datasets and models, but not across task types.
