Evals

An eval replays one step of a workflow in a world you describe: what the run started with, what the steps before it produced, and what its tools answer. Then it checks the result. Evals live on the workflow's Evals page, next to its history.

They are for the steps whose output can drift: an llm step that classifies, an agent that decides which tools to call, a javascript step with edge cases. When you or the assistant change the workflow, the evals show whether those steps still do what they should.

Side effects never happen

An eval never posts, sends, writes or deletes anything. Every tool call the step makes is recorded on the result, and:

• a tool you stub under stubs gets the stub's answer,
• a mutating tool you don't stub (Slack messages, emails, GitHub writes, artifact writes, remember) gets { ok: true, stubbed: true },
• a read-only tool you don't stub runs for real, through the workflow's egress.

The steps before the one under test don't run at all: their outputs come from the eval's steps fixtures.

Writing one

The quickest start is a real run. Open a finished step on a run's page and choose Make an eval. The form fills in that run's inputs, the outputs the step read, and what it produced. Check that the output is what the step should produce before you save.

step: classify_ticket
inputs: { team: support }
steps:
  fetch_ticket:
    output: { title: "Refund not received", body: "I was charged twice…" }
stubs:
  slack.post_message: { returns: { ok: true, ts: "1.0" } }
  zendesk.*: { fails: "Rate limited" }
expect:
  - { path: data.priority, equals: high }
  - { path: data.category, one_of: [billing, refunds] }
  - judge: Explains the priority in one sentence that cites the customer's problem

Key	What it is
step	The key of the step to replay. Required.
trigger, inputs	What the run started with, as {{ trigger.* }} and {{ inputs.* }} read them.
steps	Each upstream step's output (and status or error if it didn't succeed).
artifacts	Artifacts to hold still, by slug. Without it the step reads them live.
stubs	A tool name, or prefix.*, mapped to returns: <value> or fails: "<message the step sees>".
expect	The checks. Required.
judge_model	The model name judges use, instead of the account's default.

Checks

Check	Passes when
{ status: failed }	The step ends in that status. Without one, the step has to succeed.
{ path: …, equals: … }	The value at the path equals it exactly.
{ path: …, contains: … }	A string contains it (ignoring case), a list holds it, or a mapping has those keys and values.
{ path: …, matches: "…" }	The value matches the regular expression.
{ path: …, one_of: [ … ] }	The value is one of the list.
{ path: …, present: true }	There is a value.
judge: …	A model, given the rubric and the output, says it meets the rubric.
{ called: tool, with: { … }, times: n }	The step called the tool, with at least those arguments, that many times.
{ not_called: tool }	The step never called it.

path is a JMESPath query into the step's output, the same one builtin.json_query takes: data.priority, text, items[0].id. Leave it out to check the whole output.

When the workflow changes

An eval belongs to a step key, not a version, so it runs against whichever version you choose. Each one shows how it fits the current version:

• current: it replays as written.
• stale: the step now reads {{ steps.* }} outputs the eval has no fixture for. Add them, or make the eval again from a newer run.
• orphaned: the version has no step with that key any more.

Stale and orphaned evals aren't run, because a failure there would say nothing about the workflow.

Turn an eval off to keep it without running it. Run all runs every eval that is on against the current version.

Needs a decision

Evals ask for a person's judgement in a few places, and each one waits on the workflow's Evals page, and across every workflow under Eval decisions, until someone decides:

What	What you can do
A step worth an eval: an llm or agent step, one whose output another step's if reads or a tool sends out, one that has been failing, or one whose evals only cover runs that came back empty once a run has filled them	Have the assistant write an eval · Dismiss · Never for this step
An eval the assistant wrote	Review and accept its expectations · Discard
A stale eval	Re-capture its inputs from the latest run, keeping its stubs and checks · Edit it · Retire it
An orphaned eval	Point it at a step that exists · Retire it
Accepted evals that failed on a version	Look at the suggested fix · The eval is wrong · Ignore for this version

Find steps worth an eval looks for candidates at any time; with automatic evals on, every new version does too. A javascript step that reads nothing — no step, input, trigger or artifact — is never one: it returns what its code says, so an eval could only repeat the code.

The assistant writes evals from real runs of the step as it is now — a run from before the step last changed shows what it used to do. It starts from the recent run with the most in its output, not simply the newest, and sees how the step's other recent runs went, so it can add a second eval for an edge case that keeps happening, such as a day a site blocked the workflow. An eval it writes has to agree with the run it came from, and is sent back if its checks would also pass when the step produced nothing at all, or if it holds a fixture, trigger or inputs the step never reads. A step that runs no model did what its code does, so the assistant can't decide its output was wrong.

A javascript step goes further, since replaying it is free. Its draft is graded against the step replayed on the draft's own fixtures, so the assistant can trim a long list down to the few entries that exercise each rule. The step is also replayed with its code broken in small ways — a comparison turned around, a filter that keeps everything, a limit one off — and a draft that would let such a break through is sent back for a check that catches it.

What it writes is a draft. A draft runs only when started by hand, and joins the others once someone accepts it, because only a person knows whether an output was right. A step that runs no model has its drafts replayed straight away, and the decision says whether the draft passes; one that fails is most likely wrong itself, so it never asks for a fix to the workflow.

Suggested fixes

When evals fail, the assistant can suggest a change to the workflow that makes them pass. Before you see it, the suggestion is replayed against the evals that failed, so it shows how many it actually fixes. You can:

• Save it as a new version, which runs the evals again;
• Open it in the assistant, where it waits as an unsaved draft you can keep changing;
• say the eval is wrong, and edit the eval instead.

If the assistant thinks an eval expects the wrong thing, it says so instead of bending the workflow to fit. Nothing is ever saved without someone choosing to. With automatic evals on, a fix is suggested as soon as a pass finishes with failures, once for the same failing evals; otherwise, or for a later version that still fails them, ask for one from the decision or an accepted eval's failed result. A later version that passes those evals closes the proposal on its own.

Automatic evals

With Automatic evals on, a workflow's accepted evals run each time it changes, once the new version has settled for a couple of minutes. The same pass raises evals that no longer fit, proposes steps worth an eval, and asks the assistant for a fix when anything fails. The assistant can save several versions while it works, and only the last one is evaluated. Owners and admins turn it on per workflow, and each plan includes a number of workflows:

	Runs at once	Run history	Workspace hours a month	Workflows with automatic evals
Team	5	30 days	5	3
Scale	25	1 year	50	25
Enterprise	Negotiated	Negotiated	Negotiated	Negotiated

Moving down a plan never turns anything off: the workflows switched on most recently keep running, and the rest wait until there is room.

What evals cost

Matchers and javascript steps cost nothing. An llm or agent replay, and every judge, runs on a model like any step: your own provider's keys bill you directly, and a Murmurator AI model is metered from your credit. Each eval run stops at $0.50 of Murmurator AI. Writing evals and suggesting fixes is the assistant's work, included in your plan: it counts toward the assistant's fair use, never your credit, and has a monthly share of that allowance of its own so it can't crowd out chatting with the assistant. With automatic evals on, a fix is suggested by itself once for a set of failing evals; if a later version still fails the same evals, ask for another when you want one. AI models → Usage shows how much of both you've used.

Results and decided proposals are kept as long as your plan keeps run history.

Over MCP

get_eval_reference, list_evals, get_eval, create_eval, update_eval, run_eval, get_eval_run and list_eval_decisions work the same over MCP. An eval created over MCP counts at once; one the in-app assistant writes is a draft.