Meta-Harness: End-to-End Optimization of Model Harnesses

Takeaway

• Full trajectories, not summaries
• A coding agent does the evolving
• It’s a search problem; the hard part is credit assignment
• Code-as-representation is a free regularizer

Remaining work

• Credit assignment stays qualitative (no per-component score), and no guarantee each round improves.
• Large, open-ended search space explored under a tight budget — running each candidate on the frozen model is the compute bottleneck.
• Generalizes across datasets and models, but not across task types.