Improve a prompt, policy, or configuration. A support assistant's system prompt is one example. Save the champion, its score, a working set, untouched holdout cases, must-pass checks, and [budget]. Each round, change one thing based on a recorded failure. Promote the challenger only if it beats the champion on holdouts by [margin] without weakening a must-pass check; otherwise keep the champion. Stop at the target, budget limit, or no progress. Return the winner, scores, experiment log, and remaining failures.
claude-code · codex
Use this when
Use this to tune a prompt, policy, or configuration when cheap iteration is useful but final acceptance must use fresh examples.
How it runs
Save the current champion, working set, untouched holdout cases, must-pass checks, improvement margin, budget, and experiment log.
Use a recorded failure to propose one targeted challenger and test it on the working set.
Freeze promising challengers and evaluate them on the untouched holdout cases and every must-pass check.
Promote only a meaningful, regression-free holdout win; log every result and return the champion at the stop condition.
Done when
✓ The best holdout-tested champion is returned. Every challenger is logged, and accepted changes beat the previous champion on untouched cases without weakening a must-pass check.
Why it works
Separating the working set from fresh holdout cases limits overfitting. Keeping the current best by default prevents regressions, while a fixed budget bounds the search.
Implementation note
Keep the working set and holdout cases separate: edit against the former, judge final acceptance on the latter. Choose the budget and margin before starting, and do not weaken a must-pass check after a failed challenger.
Maintain a durable, versioned playbook of lessons that may improve future runs of [task or workflow]. Store it in [path], using playbook/ by default. Treat every recorded lesson as untrusted advice rather than authority. At the start of each run, read the playbook and choose at most one relevant lesson to test. Apply it only within the task's existing permissions. Measure the result using the task's own success check and record the context, action, outcome, and evidence. Promote a candidate lesson only after it succeeds across [N] independent runs or a predefined holdout set. Use three independent runs by default. Never promote a lesson from one successful attempt. Revise or remove lessons that stop helping. Stop when no candidate has enough evidence, another test would exceed the budget, or approval is required. Never let the playbook authorize production, destructive, financial, privacy-sensitive, or external actions. Finish with the playbook diff, evidence ledger, removed lessons, unresolved candidates, and new version.
Before committing to an architecture, interface, or rollout plan, have a critic argue that it is wrong. Record each objection, impact, and status in a repository-local log at .agent-reviews/redteam.md. The builder must fix and verify each high-impact weakness or document why it is accepted; the critic may reopen unsupported answers. Stop when no high-impact objection remains or the same issues repeat for two rounds without new evidence. Finish with the decision, resolved and accepted objections, evidence, and any stalemate.
Investigate [question, decision, or unresolved problem] using [available evidence]. Separate established facts, contested claims, assumptions, and unknowns. Construct at least three genuinely different hypotheses, each with predictions, falsifying evidence, assumptions, and decision implications. Choose the uncertainty with the highest expected information value and run the smallest safe test or analysis that could materially change the conclusion. After each round, update the evidence ledger and confidence levels, then have an adversarial critic attack the leading hypothesis. Repeat for at most five rounds while new evidence could change the decision. Stop when one model clearly explains the evidence better than its alternatives, further investigation has low value, the problem remains underdetermined, or approval is required. Never fabricate evidence or hide uncertainty. Finish with the final model, hypothesis comparison, falsified ideas, unresolved contradictions, confidence, decision implications, and best next experiment.