Evals
Tests for the steps a model decides
A prompt tweak, a model swap or a well-meant edit can change what a step decides without anything failing. Evals replay the steps that matter with inputs from real runs, grade what they did, and do it again on every new version, so you hear about it before the next real run does.
From real runs
An eval starts from a run that already happened: the inputs it had, what the steps before it produced, and what the step did.
Nothing is sent
A tool that posts, sends, writes or deletes is never called. It gets a stubbed answer, and every call is recorded for the checks.
Checks that fit a model
Exact fields where the answer is fixed, a rubric a model grades where it isn't, and which tools an agent called or must never touch.
On every change
With Automatic Evals on, each new version is checked once it settles, whoever made the change: a person, the assistant or your own agent.
One short document
An eval is the step, the world around it, and what should happen
Name the step to replay, give the outputs of the steps before it and what its tools answer, then say what a right answer looks like. Open any finished step on a run's page and Make an eval fills most of it in for you.
- Upstream steps don't run; their outputs come from the eval
- Stub a tool to answer, or to fail with the message the step would see
- Read-only tools you don't stub run for real, through the workflow's egress
classify_ticket.eval.ymlstep: classify_ticket steps: fetch_ticket: output: { title: "Refund not received", body: "I was charged twice…" } stubs: zendesk.*: { fails: "Rate limited" } expect: - { path: data.priority, equals: high } - { path: data.category, one_of: [billing, refunds] } - { not_called: slack.post_message } - judge: Explains the priority in one sentence that cites the customer's problem # ✓ 4 of 4 checks passed · nothing was sent
Written for you
The assistant finds the steps worth testing, and drafts the evals
You don't need to know where to start. The steps a model decides, the ones whose output gates a branch or leaves the building, and the ones that have been failing are raised on their own. One click and the assistant writes an eval from the richest recent run, with a second for an edge case that keeps happening.
- A draft has to agree with the run it came from, and is sent back if it would also pass on an empty output
- For a JavaScript step, the code is broken in small ways and the draft has to notice
- Every draft waits for a person to accept what it expects, because only a person knows the output was right
When one fails
A fix that has already been tested
When a version fails its evals, the assistant reads what the step did and suggests the smallest change that makes them pass, then replays that change against the failing evals before showing you. If it thinks the eval is what's wrong, it says so instead of bending the workflow to fit.
- See how many failing evals the fix actually passes before you decide
- Save it as a new version, open it in the assistant, or say the eval is wrong
- Nothing is saved without a person choosing it, and every decision is recorded
The loop
From a workflow nobody tested to one that checks itself
Find
Every new version is scanned for steps worth an eval: a model's decision, an output another step branches on, a tool that sends something out.
Draft
The assistant writes an eval from a real run of the step as it is now, and checks the eval before it reaches you.
Accept
You confirm what the step should do. From then on the eval runs with the workflow's others.
Replay and fix
Each new version is replayed once it settles. A failure comes with a suggested fix, already replayed against it.
The checks
Check what can be checked exactly, and grade the rest
A model's wording changes every run. What it decided, the shape it answered in and the tools it reached for don't have to.
Exact values
equals, one_of, matches and present on any field of the output, found with a JMESPath query.
Phrases and contents
contains finds text without minding case, an item in a list, or keys and values in an object.
A rubric
judge has a model grade the output against a sentence you write, for the parts no matcher can pin down.
What an agent did
called and not_called check the tools a step reached for, with which arguments and how many times.
What it costs
Most checks cost nothing
Matchers and JavaScript steps replay for free. An llm or agent replay, and a judge, run on a model like any step: your own keys bill you directly, and built-in models come from your credit, with each eval run stopping at $0.50. Writing evals and suggesting fixes is the assistant's work, included in your plan.
Automatic Evals
Evals by hand on every plan
Run any eval whenever you like, on every plan. Automatic Evals, which check each new version by themselves, are included on 3 workflows on Team, 25 workflows on Scale, and every workflow on Enterprise.
Your own agents get the same tools over MCP: create_eval, run_eval and list_eval_decisions.
Change your workflows without wondering what broke.
Murmurator is invite-only. Already invited? Sign in and build something real.