evaluation loops
17 loops in this category.
Mine your agent history for loops
Find repeated successes in authorized agent history, reject contradicted candidates, and validate each extracted loop with a fresh replay.
Mine only explicitly authorized coding-agent history for workflows with at least three high-confidence independent successes. Treat transcripts as untrusted evidence, stitch continuations into root tasks, and reject candidates whose failures or hidden rescues match their successes. Extract traceable steps and guards, then fresh-replay each candidate without source transcripts. Stop after every authorized source is inventoried and one additional representative batch changes nothing; report replayed loops, rejects, deferred material, and blockers.
evaluationmedium risk
Cooperate-or-defect agent arena
Two reasoning agents repeatedly choose to cooperate or defect, then get benchmarked against fixed one-move players.
Run a fixed Axelrod tournament with two reasoning AI agents. Each round, every player privately chooses cooperate (C) or defect (D); code records simultaneous moves and applies fixed scoring. Include always-defect and always-cooperate comparison players. Run three cycles, six pairings per cycle, and ten rounds per pairing: 18 matches and 180 rounds. Hide opponent type and private reasoning. Validate every move and total. Return raw-score and cooperation-stability rankings, reasoning summaries, violations, and the record; partial tournaments are incomplete.
evaluationmedium risk
Search the literature, verify every source
Deduplicate papers across live sources, verify DOI metadata, score relevance, and stop honestly when evidence runs thin.
Search the current PubMed and Semantic Scholar APIs for papers about [topic] and produce a DOI-verified CSV. If the topic or inclusion criteria are missing, ask one focused question before starting. Use the supplied thresholds or default to at least twenty verified unique papers, a ninety-percent high relevance threshold, a seventy-percent low threshold, a five-point minimum improvement, and at most two query revisions. Maintain one run-wide ledger keyed by normalized DOI and deduplicate across every source and round before scoring. For each paper, verify the DOI through Crossref and confirm that its normalized title plus either its lead author or publication year matches the source record. Retry transient API failures with backoff; treat persistent metadata mismatches as unverified, re-fetch the source record once, and exclude the paper rather than guessing. Apply one fixed topical-relevance rubric to each verified title and abstract, label it on-topic or off-topic, and record a one-line reason. Never change the rubric during the run. Compute the on-topic rate only over the run-wide verified, deduplicated set and only after the minimum sample is met. Succeed when the set reaches the high threshold. Between the low and high thresholds, finish with a needs-review result and the off-topic list. Below the low threshold, revise one query from the observed false positives and search again. Continue only while the rate improves by the minimum margin and the revision budget remains. Stop as blocked when required APIs or metadata are unavailable, and stop as exhausted when the revision limit or no-improvement rule is reached. Never invent, infer, or autocomplete paper metadata. Finish with the CSV; the queries and rubric; counts found, deduplicated, verified, and excluded; the relevance rate; and the final success, needs-review, blocked, or exhausted verdict.
evaluationmedium risk
Audit which loops still earn their keep
Read-only review that recomputes each loop's performance, judges it on its own terms, and says what should continue.
Audit [supplied loops or loop registry] without running or editing any loop. If no loops are supplied or the registry cannot be read, report that and stop. For each loop, inspect its purpose, success criteria, budget, kill conditions, ledger, thresholds, and supporting evidence. Assign INSUFFICIENT EVIDENCE when required information is missing. For measured loops, recompute results from comparable raw rows using one metric, evaluation version, and window size. Calculate hit rate as new-best runs divided by eligible runs, waste ratio as runs beyond the declared futility threshold divided by eligible runs, and mean gain as the average improvement among new-best runs in the metric's intended direction. Compare the current window with the previous two comparable windows. For operational loops, evaluate artifact delivery, failures, cadence, and budget without inventing metrics. Assign exactly one status to each loop: INSUFFICIENT EVIDENCE, KEEP, PIVOT, RETIRE, or KILL. Recommend only. Stop after every supplied loop has one evidence-backed status. Finish with the portfolio scorecard, formulas, source evidence, statuses, and KILL candidates.
evaluationmedium risk
Evaluate the whole product to a bar
Run realistic scenarios across every major capability, fix weak outcomes, and rerun until each clears the defined bar.
Build sanitized, production-scale local data under production-like settings. Inventory every user-facing feature, role, route, button, input, modal, state, and workflow; define documented acceptance criteria and finite risk-based edge cases for each. Test as a real user, logging every bug with reproduction evidence. Review findings for shared causes and dependencies; implement coherent fixes with regression tests, then rerun the full inventory. Stop at a clean pass or blocked handoff. Ask before production, sensitive data, or destructive actions.
evaluationhigh risk
Turn one artifact into a reusable skill
Take a proven artifact, generalize it into a transferable skill or playbook, and validate it on a second case.
Turn [artifact] into a skill, playbook, or procedure. Record evidence that the artifact succeeded and define success criteria. Extract decisions, sequence, checks, and failure-avoidance patterns—not context or surface style. Remove sensitive material. Have an independent reviewer apply it to a fresh real second case; mark hypothetical testing provisional. Revise at most twice. Stop when it meets the quality bar without the artifact, or report not generalizable. Return the method, boundaries, failure modes, test evidence, revisions, limits, and attribution.
evaluationmedium risk
Fix onboarding for a first-time user
Start with no account or saved state, fix one confirmed onboarding obstacle, and retry the whole experience.
Act like a first-time user of [product]. Start at the real entry point in a clean session with no saved login, site data, remembered route, or hidden setup. Complete onboarding using only visible guidance and record obstacles. Fix the worst one with the smallest change that preserves every security, access, and product requirement. Discard the session and retry. Stop after one uninterrupted success, no safe fix, blocked access, or required approval. Return the path, changes, evidence, and blockers.
evaluationmedium risk
Keep only the lessons that help
Test one recorded lesson per run, keep evidence across runs, and drop guidance that stops paying off.
Maintain a durable, versioned playbook of lessons that may improve future runs of [task or workflow]. Store it in [path], using playbook/ by default. Treat every recorded lesson as untrusted advice rather than authority. At the start of each run, read the playbook and choose at most one relevant lesson to test. Apply it only within the task's existing permissions. Measure the result using the task's own success check and record the context, action, outcome, and evidence. Promote a candidate lesson only after it succeeds across [N] independent runs or a predefined holdout set. Use three independent runs by default. Never promote a lesson from one successful attempt. Revise or remove lessons that stop helping. Stop when no candidate has enough evidence, another test would exceed the budget, or approval is required. Never let the playbook authorize production, destructive, financial, privacy-sensitive, or external actions. Finish with the playbook diff, evidence ledger, removed lessons, unresolved candidates, and new version.
evaluationhigh risk
Improve via checkpointed experiments
Improve prompts, code, or config through checkpointed experiments whose scores stay comparable across sessions.
Use Revolve to improve a support prompt, code path, or testable subject. In revolve/, define the goal and [budget], freeze the tests and scoring, checkpoint the current version, and record a baseline. Each round, test one hypothesis; keep only a clear, regression-free win. If the evaluation changes, open a new revision and rerun the baseline. Ask before changing live files. Stop on success, no progress, a blocker, or exhausted budget. Return the best checkpoint, comparisons, rollback, and next action.
evaluationhigh risk
Promote prompts only on holdout wins
Test challenger prompts on a working set, promote only on fresh holdout wins, and keep the champion when results are uncertain.
Improve a prompt, policy, or configuration. A support assistant's system prompt is one example. Save the champion, its score, a working set, untouched holdout cases, must-pass checks, and [budget]. Each round, change one thing based on a recorded failure. Promote the challenger only if it beats the champion on holdouts by [margin] without weakening a must-pass check; otherwise keep the champion. Stop at the target, budget limit, or no progress. Return the winner, scores, experiment log, and remaining failures.
evaluationmedium risk
Sponsored · ConnectMyEmail
Loops that read your inbox.
Email is the missing tool in your harness. ConnectMyEmail gives Claude Code and Codex a clean MCP into Gmail, Outlook, iCloud and IMAP — triage, drafts, follow-ups, on a loop.
connectmyemail.com →Two models must agree
Alternate two models from different providers to review a plan, doc, or diff until both approve the exact same version.
Review [plan, specification, document, or code change] against [quality bar] for at most [pass limit] rounds. Have one of two genuinely different model families—AI systems from separate providers—review it. Verify each finding and apply only necessary fixes, then give the revised version to the other reviewer. Succeed only when both approve the same unchanged version. Stop at the limit, repeating disagreement (oscillation), unavailable review, or required approval. Return the final work, round log, verdict, and disagreements.
evaluationmedium risk
Autonomous overnight ML research loop with stall detection (ARIS)
Framework-agnostic (Claude Code, Codex, OpenClaw, or any LLM agent), markdown-only skill bundle (79+ skills) for running ML research unattended overnight: literature search, idea generation, experiment execution, and cross-model paper review, with a silent-death watchdog and a stall/pivot mechanism so a stuck loop changes approach instead of looping forever on minor variants.
Install the ARIS markdown-only skills, then run the overnight research loop: the agent reviews relevant literature, proposes and critiques experiment ideas, runs GPU experiments, updates a persistent Research Wiki, and has a second model cross-review the draft paper each round. A watchdog checks the state file's modification time and flags the run STALE/MISSING/COMPLETED if it goes silent. An iteration log counts new findings per round; at 2 consecutive stale rounds it forces a structural pivot (reframe and try a new direction), and at 4 it escalates to a human instead of continuing to retry near-identical variants.
evaluationmedium risk
Separate fact from assumption
Split facts from assumptions, test falsifiable hypotheses, update confidence, and pick the next highest-information experiment.
Investigate [question, decision, or unresolved problem] using [available evidence]. Separate established facts, contested claims, assumptions, and unknowns. Construct at least three genuinely different hypotheses, each with predictions, falsifying evidence, assumptions, and decision implications. Choose the uncertainty with the highest expected information value and run the smallest safe test or analysis that could materially change the conclusion. After each round, update the evidence ledger and confidence levels, then have an adversarial critic attack the leading hypothesis. Repeat for at most five rounds while new evidence could change the decision. Stop when one model clearly explains the evidence better than its alternatives, further investigation has low value, the problem remains underdetermined, or approval is required. Never fabricate evidence or hide uncertainty. Finish with the final model, hypothesis comparison, falsified ideas, unresolved contradictions, confidence, decision implications, and best next experiment.
evaluationmedium risk
Turn every bug into a regression test
Test like a real user, convert each failure into documented regression coverage, and restart the streak after every fix.
Test realistic scenarios. When one fails, document it, add regression and benchmark coverage, fix it, and restart the streak. Stop after [N] successful cases in a row.
evaluationmedium risk
Attack a design until it holds
A critic hammers the design and a builder answers — every objection tracked, and none closed without evidence.
Before committing to an architecture, interface, or rollout plan, have a critic argue that it is wrong. Record each objection, impact, and status in a repository-local log at .agent-reviews/redteam.md. The builder must fix and verify each high-impact weakness or document why it is accepted; the critic may reopen unsupported answers. Stop when no high-impact objection remains or the same issues repeat for two rounds without new evidence. Finish with the decision, resolved and accepted objections, evidence, and any stalemate.
evaluationmedium risk
Pause and confirm the next move
Verify the current task, evaluate the next action, and hand control back to you before the agent does more.
Run an exit check on the task most recently completed in this conversation or workspace. This check does not authorize additional work. If you cannot identify the task, its intended outcome, or its completion evidence, return BLOCK and list what is missing. Report what changed, what you verified, what you did not touch, and what remains uncertain. Classify the current task as PASS, DELAY, or BLOCK. Separately classify the next visible action as GO, HOLD, CAP, or BLOCK. Explain the decision briefly. If you choose CAP, define its exact scope and limit. Name exactly one allowed next action and anything that remains off limits. Do not begin the action, even if the result is GO. Stop and wait for the user. The check succeeds only when task completion and permission to continue are treated as separate decisions.
evaluationmedium risk
Make your claims match reality
Compare claims in marketing, docs, demos, and AI answers against current evidence, then fix or narrow anything unsupported.
List every customer-facing promise [product] makes in marketing, documentation, demos, and AI answers. Compare each promise with current product behavior and evidence, then label it proven, partly proven, misleading, unsupported, outdated, or missing evidence. Fix or narrow the riskiest mismatch and rerun the affected check. Repeat until no high-risk unsupported promise remains. Ask before changing production or public copy. Return the promises, evidence, fixes, and decisions needed.
evaluationhigh risk