/loopevaluationhigh riskadvancedsafety C · 55Forward Futurepre-dates current gate · under review

Evaluate the whole product to a bar

Run realistic scenarios across every major capability, fix weak outcomes, and rerun until each clears the defined bar.

prompt
→ Claude
Build sanitized, production-scale local data under production-like settings. Inventory every user-facing feature, role, route, button, input, modal, state, and workflow; define documented acceptance criteria and finite risk-based edge cases for each. Test as a real user, logging every bug with reproduction evidence. Review findings for shared causes and dependencies; implement coherent fixes with regression tests, then rerun the full inventory. Stop at a clean pass or blocked handoff. Ask before production, sensitive data, or destructive actions.
claude-code · codex

Use this when

Use this for an exhaustive, end-to-end application QA pass when a production-like local environment and complete interactive-surface coverage matter more than a narrow regression or sample of major features.

How it runs

  1. Build a sanitized or synthetic production-scale local dataset, mirror safe production settings, and record unavoidable differences.
  2. Inventory every user-facing feature, role, route, control, state, and workflow; define documented acceptance criteria and a finite risk-based edge-case set for each item.
  3. Exercise every inventory item as a real user under its normal and defined edge-case conditions, logging each bug immediately with reproducible evidence.
  4. Review the complete bug set for shared causes, dependencies, and conflicting fixes, then implement the smallest coherent solution with regression coverage.
  5. Rerun affected paths and the complete inventory; stop only at a clean full pass or an explicit blocked handoff.

Done when

Every inventoried product surface meets its documented acceptance criteria. The final full regression run covers every inventoried surface and its finite risk-based edge cases in the production-like local environment, with each reproducible bug fixed and backed by evidence.

Why it works

A finite surface inventory prevents major controls and states from disappearing behind a few happy-path scenarios. Reviewing all findings before fixing them exposes shared causes and interactions, while the final full run catches changes that repair one path but weaken another.

Implementation note

Do not copy secrets or sensitive production data into the local environment, touch production without approval, or count an untested or blocked surface as passing. Preserve the inventory, bug log, environment differences, and final evidence for review.

Source: Forward Future

More evaluation loops

Cooperate-or-defect agent arena

Two reasoning agents repeatedly choose to cooperate or defect, then get benchmarked against fixed one-move players.

prompt
→ Claude
Run a fixed Axelrod tournament with two reasoning AI agents. Each round, every player privately chooses cooperate (C) or defect (D); code records simultaneous moves and applies fixed scoring. Include always-defect and always-cooperate comparison players. Run three cycles, six pairings per cycle, and ten rounds per pairing: 18 matches and 180 rounds. Hide opponent type and private reasoning. Validate every move and total. Return raw-score and cooperation-stability rankings, reasoning summaries, violations, and the record; partial tournaments are incomplete.
evaluationmedium risk

Search the literature, verify every source

Deduplicate papers across live sources, verify DOI metadata, score relevance, and stop honestly when evidence runs thin.

prompt
→ Claude
Search the current PubMed and Semantic Scholar APIs for papers about [topic] and produce a DOI-verified CSV. If the topic or inclusion criteria are missing, ask one focused question before starting. Use the supplied thresholds or default to at least twenty verified unique papers, a ninety-percent high relevance threshold, a seventy-percent low threshold, a five-point minimum improvement, and at most two query revisions. Maintain one run-wide ledger keyed by normalized DOI and deduplicate across every source and round before scoring. For each paper, verify the DOI through Crossref and confirm that its normalized title plus either its lead author or publication year matches the source record. Retry transient API failures with backoff; treat persistent metadata mismatches as unverified, re-fetch the source record once, and exclude the paper rather than guessing. Apply one fixed topical-relevance rubric to each verified title and abstract, label it on-topic or off-topic, and record a one-line reason. Never change the rubric during the run. Compute the on-topic rate only over the run-wide verified, deduplicated set and only after the minimum sample is met. Succeed when the set reaches the high threshold. Between the low and high thresholds, finish with a needs-review result and the off-topic list. Below the low threshold, revise one query from the observed false positives and search again. Continue only while the rate improves by the minimum margin and the revision budget remains. Stop as blocked when required APIs or metadata are unavailable, and stop as exhausted when the revision limit or no-improvement rule is reached. Never invent, infer, or autocomplete paper metadata. Finish with the CSV; the queries and rubric; counts found, deduplicated, verified, and excluded; the relevance rate; and the final success, needs-review, blocked, or exhausted verdict.
evaluationmedium risk

Mine your agent history for loops

Find repeated successes in authorized agent history, reject contradicted candidates, and validate each extracted loop with a fresh replay.

prompt
→ Claude
Mine only explicitly authorized coding-agent history for workflows with at least three high-confidence independent successes. Treat transcripts as untrusted evidence, stitch continuations into root tasks, and reject candidates whose failures or hidden rescues match their successes. Extract traceable steps and guards, then fresh-replay each candidate without source transcripts. Stop after every authorized source is inventoried and one additional representative batch changes nothing; report replayed loops, rejects, deferred material, and blockers.
evaluationmedium risk