Why Your Loop Got an F: How Agent-Loop Grading Actually Works

Why Your Loop Got an F: How Agent-Loop Grading Actually Works

We ran fifteen published Ralph-style loops — real ones, read at the source — through the same A–F rubric the directory uses. Exactly one climbed above an F — a single D. Every other executable loop scored F, with one prompt-only entry set aside as an ungraded methodology. The most-starred entry in the set — Anthropic's own ralph-wiggum plugin, sitting inside a 137,000-star repository — scored an F.

That is not the grader misfiring. That is the grader doing the one job a safety grade exists to do: refuse to grade on reputation. This is how it reaches that verdict, and how you move a loop off the floor.

The grader doesn't know who wrote it

The rubric is default-fail and provenance-blind. It does not read stars, author, or vibe. It reads the loop's description for concrete evidence of two things: a hard stop the model cannot fudge, and a completion check the model cannot fake. Absent either, it fails by default — an unclear loop is a reject, not a maybe.

That design has a consequence people find funny right up until they think about it: if a famous logo could buy a passing grade, the grade would be worthless. A safety gate that trusts the byline is not a gate. So the highest-star loop in the set gets no discount, and the 0-star loop with a real verifier and a mandatory budget outscores it. That inversion is the point.

Two ceilings do most of the work

The grade is a weighted score across seven dimensions — boundedness, verifiability, durability, recovery, safety, observability, usability — but two hard caps dominate the outcome. A cap is a ceiling: it can only pull a grade down, never up.

Nearly every Ralph loop ever published is exactly one of the first two shapes: a while true that stops when the model says so. So they cluster at D and F before the seven dimensions are even tallied — and because a terse description also leaves verifiability and durability near zero, the raw score lands under the F line too. Both forces push the same direction.

Why Anthropic's loop is an F — and it's the honest grade

Anthropic's ralph-wiggum plugin is a deliberately minimal teaching device, not a production harness, and its own files say so:

• max_iterations defaults to 0 = unlimited. The setup script prints a warning that the loop "runs infinitely" and can't be stopped by normal means. → unbounded_no_cap.
• Completion is a model-emitted <promise>COMPLETE</promise> string. The Stop hook runs no tests, lint, or build — it string-matches the transcript. Persistence, not correctness. → completion_promise_only.

Two independent caps, an F. And Anthropic would not argue: the plugin ships with that warning banner precisely because it is a demonstration of the mechanism, not something to point at production and walk away from. Read the F as a caption, not an insult: unbounded, self-attested completion, no enforced check — do not run unattended until you add a cap and a real gate.

Which is also the path up. The grade is not a life sentence:

• Set --max-iterations (or any wall-clock ceiling) and unbounded_no_cap stops firing — the D ceiling lifts.
• Put a machine check in front of the promise — a test suite that must pass before "done" counts — and completion_promise_only lifts too.

It's an F as shipped by default, not an F forever. The famous entry landing on the floor is the grader passing its own test. The alarming result would have been a B for the logo.

The listing floor: the grade reflects what you wrote, not what you built

This is the finding that surprises people most — and the one worth changing how you write.

The grader reads a listing — the description you submit — not your ten-thousand-line source tree. It keys on concrete tokens: npm test, pytest, exit 0, state.json, max_iterations 50. If your prose paraphrases those instead of naming them, the signal never fires.

The starkest case in our corpus was the single most rigorously engineered loop in the set: a fresh-process orchestrator with a five-defense verification stack, an independent cross-backend judge, held-out tests the implementer can't see, and a rule that it refuses to start unless every budget cap is set. Its dimension line:

boundedness 19 · verifiability 0 · durability 0 · recovery 5 · safety 6 · observability 0  →  F

Verifiability: zero. A five-layer-verified loop scored nothing on verification — because its description said "runs held-out tests," and the grader's test signal wants a named runner or an explicit tests must pass / exit 0, which "held-out tests" doesn't match. Same for durability: the state files existed in the code but weren't named in the paragraph.

That F is not a verdict on the software. It's a verdict on the sentence. The gap between listing-grade and real maturity is the one lever worth pulling here: a genuinely excellent loop floors to F if its listing doesn't state its caps and its gate in words the grader can key on. Grade the description you'd actually submit, then fix the description, not just the code.

Two rules baked into the score

The rubric encodes two opinions that explain most surprises:

• Persistence is not correctness. A loop that runs until the model announces it's done has proven stamina, not results. That belief is the entire reason a model-promise stop is capped at C: no matter how much else it does right, an unverifiable finish line ceilings it.
• A retry is not self-healing. Re-running the same prompt after a failure earns almost nothing on the recovery dimension. Recovery credit requires an observable strategy change — a rollback, a fresh-context diagnosis, a decomposition, an escalation to a human. A loop whose marketing says "self-healing" but whose code just loops again gets the claim flagged, not rewarded.

If a grade feels harsh, it's usually one of these two rules refusing to give credit for effort that can't be checked.

How to earn a real grade

You raise a grade by making the invisible legible. Same loop, two descriptions:

Floors to F: > A control-plane loop that iterates until the work is done, running tests each pass and committing as it goes. Powered by a fresh process each iteration.

Nothing there is a token the grader can trust. "Until the work is done" is a model promise. "Running tests" names no runner. No cap appears at all.

Clears the caps: > Fresh codex exec each iteration. Stops after 15 iterations or an 8-hour wall-clock ceiling, whichever first. Each pass runs npm test and npm run build; the loop only advances on green — a failing suite blocks completion. Reads prd.json / progress.txt for state; commits per task. On failure it classifies the error and regenerates the plan; escalates to a human after 3 attempts.

Every claim is now a signal: a loop bound (lifts unbounded_no_cap), an executed and gated suite (lifts completion_promise_only and scores verifiability), named state files (durability), a real strategy change (recovery). Nothing was invented — the second version just names what the first one hid.

The checklist, in order of leverage:

1. Name a loop-level cap. An iteration/cycle count or a wall-clock ceiling. Not a per-call timeout — that caps one turn, not the run. 2. Name a gate the model can't fake. A concrete test/build command that must pass before completion counts. Bonus: an independent reviewer that didn't write the code. 3. Name your state. The files that survive between runs (prd.json, tasks.md, state.json). 4. Name real recovery. Rollback, replan, escalate — not "retries." 5. Cut the destructive commands. Anything matching rm -rf, force-push, or curl | sh is an automatic F, no matter what else is true.

The grade is a mirror

Read an F as a question, not a sentence: what can't this loop prove yet? Usually the answer is one of two things — it has no hard stop, or its finish line is the model's own word — and both are fixable in an afternoon. Sometimes the loop is already sound and only its description is starving the grader; that's fixable in a paragraph.

So grade the loop you'd actually ship, in the words you'd actually submit — then fix whichever half is weak and grade it again.

Paste your loop into /grade before you publish it or leave it running unattended. It's free, it takes a paragraph, and it names the exact caps you're tripping and what lifts each one. Then open the directory: every loop there already carries this verdict, so you can read a dozen real ones at a glance — which ship with a genuine cap and a genuine gate, and which are one unbounded night from trouble.

Still building the machine under the grade? The Anatomy of an Agent Loop is the build order, and Loops Gone Wrong is what the caps are protecting you from.