Skip to main content

Write an Eval Case Brief Before You Score the Agent Run

Updated 7 min readMarkdown

By AgentOps Lab

You already paste failures, misses, and “it should have done X” notes into an agent. What still fails is asking for a score, a pass rate, or a verdict before anyone agrees what behavior is under test. A messy trace goes into a blank chat; a confident grade comes out; and only then do you notice the inputs were unsanitized, the expected behavior was assumed, and the number was invented.

This post is an eval-case-brief ritual for people who already run AI agents on systems they own or are authorized to test. It is not a SOTA or benchmark guide, not an income promise, and not a jailbreak, exploit, or “grade this model against a public exam” kit.

You will leave with a brief shape, a 15-minute drill, and a pointer to other live AgentOps Lab brief rituals. The Agent Evaluation pack store listing is not up yet, so this post does not send you to a checkout page that is still unpublished.


The symptom: the blank-chat tax

Typical week:

  1. Monday — you dump a failed run into chat and ask, “how did this score?”
  2. Tuesday — the agent invents expected output and a percentage nobody measured.
  3. Wednesday — three behaviors are hiding in one “case,” so a pass on one looks like a pass on all.
  4. Thursday — a production trace still contains a token, an email, or a customer record.
  5. Friday — you redo the eval because nobody wrote down the behavior under test, the pass/fail signals, or the sanitization path.

That is not “the model being dumb.” It is missing evaluation-ops. A reusable case needs a job description the agent can follow: restate the failure or requirement → name one behavior under test → sanitize inputs → write expected behavior you can observe → define pass and fail signals → flag what is out of scope → say whether the source is an owned failure, a requirement, or an owned trace.

Skip that ritual and every later step — a golden set, a rubric, or a regression re-run — is guessing.


Brief shape you can paste into a skill

Restated source (1 sentence):
Reported facts vs assumptions (labeled):
Authorization note:
Sanitization notes:
Behavior under test:
Setup / preconditions:
Inputs (sanitized):
Expected behavior:
Pass signals:
Fail signals:
Out-of-scope / do-not-score:
Source type (owned failure / requirement / owned trace):
Next action:
Unknowns:

Rule: the agent must fill one behavior under test, sanitized inputs, and pass/fail signals before it offers a score. If those are empty, the grade is theater. If you are not authorized to test the system, or the paste cannot be sanitized, stop — do not keep the raw log as the case, and do not import someone else’s benchmark or exam.


Five fields people skip (and why they matter)

One behavior under test. “Check that it follows the policy and formats the JSON and calls the right tool” is three cases. Park the extras. A usable behavior sounds like something you could watch fail again on purpose, without inventing a second scenario.

Sanitized inputs. Tokens, passwords, private emails, and raw production customer logs do not belong in a reusable case. If the paste cannot be redacted and still make sense, it is not reusable yet. Write that down and stop rather than smuggling the raw trace forward.

Expected behavior, written as observe-able. Prefer “the agent should X and should not Y” over a vague “be good” or a gold answer copied from a public exam. If you never stated the expected output, mark UNKNOWN — do not let the agent invent one so it can award a score.

Pass and fail signals, plus out-of-scope. Signals should be checkable without a fake percentage or a leaderboard. Adjacent behaviors, other agents, and unauthorized systems belong in do-not-score, not in a quiet fail.

Facts vs assumptions, plus authorization. A quoted miss in the trace is a fact. “This model is generally weak at tools” is an assumption and should be labeled. Only evaluate agents or systems you own or are authorized to test. If that is unclear, the brief should stop and ask, not proceed as if rights exist.

A next action sits beside these: add-to-golden-set, classify-failure, write-rubric, park, or stop. One case, one next action. The agent drafts; you remain the reviewer of record.


15-minute drill (use a sanitized failure or requirement)

  1. Pick one failure, miss, or requirement from an agent or system you own or are authorized to test. Strip names, emails, tokens, secrets, and raw customer identifiers.
  2. If you are not authorized to test the system, or the paste cannot be sanitized, stop this drill. Do not build a case from a copied benchmark or exam.
  3. Paste the brief shape (or an eval-case-brief skill file, if you have one loaded) and the sanitized source into your agent.
  4. Require one behavior under test, sanitized inputs, and pass/fail signals before any score, pass rate, or “this would be a 7/10.”
  5. Accept one next action. If the agent invents expected output, imports a public benchmark, or prints a leaderboard, reject that output.

Success looks like a one-screen case you could hand to another reviewer — not a score the agent minted for you.


Hard rules worth keeping in the skill

  • Only evaluate agents or systems you own or are authorized to test
  • Sanitize PII and secrets; do not dump raw production customer logs into reusable evals
  • Do not scrape or copy others’ benchmarks, exams, or copyrighted datasets
  • Do not invent eval scores or fake leaderboards
  • No accuracy, SOTA, revenue, or job-replacement claims
  • You advise; the user owns authorization, privacy, and how they use the outputs

These rules are the difference between a reusable case and a confident, unusable grade.


What this ritual is not

It will not make your agent state-of-the-art, pass a public benchmark, or replace a reviewer. Files and rituals do not promise that.

It is not a license to evaluate systems you have no right to test, to paste unsanitized production logs, or to copy someone else’s exam set.

It is not a jailbreak, exploit, or attack-procedure kit. A brief that describes expected behavior is not a red-team script.

It is not a substitute for judgment. A clear case still needs you to decide whether to add it to a set, how to score a later run, and when to park the work.


Where to go next

The same brief-before-blank-chat idea shows up in other AgentOps Lab rituals that are already listed. If you review pull requests more often than you score agent runs, the live Coding Agent Ops teaser walks a PR-review brief (intent, blast radius, risks, test gaps) — that is a merge-safety ritual, not an agent-evaluation pack:

→ Coding Agent Ops free teaser on itch

Catalog and other live packs:

→ AgentOps Lab

A full Agent Evaluation Ops pack (eval case briefs plus golden-set scaffolds, failure taxonomies, scoring rubrics, regression-eval plans, tool-permission briefs, and weekly agent-run retros) is being prepared. This article does not link a store or product page that is not live yet. When that listing exists, it will appear on the AgentOps Lab hub.

No accuracy, SOTA, benchmark, or earnings promises. Authorized systems only. Do not invent scores. Sanitize traces. Files only; you remain the reviewer of record.


AgentOps Lab builds installable ops skills for AI agents. Deep links only — we do not ask you to trust anonymous store roots.

About the author

  • AgentOps Lab ships ready-to-run skills, prompts, and workflows. Browse free teasers → paid packs: https://agentops-lab.netlify.app/ · No fake earnings claims.

No comments yet. Be the first to comment.