Evaluation
Evaluators: model judges and code checks
The evaluator templates (Correctness, Groundedness, Relevance and more), code checks, knowledge search metrics and pairwise judges: what each needs, how it scores, which model judges, and Try it.
Reference4 min readUpdated 31 October 2026
An evaluator scores one reply from 0 to 1 and says whether it passed. Create one under Evaluate → Evaluators → New evaluator: start from a template, then change anything before you save it. Each evaluator has a Key, such as correctness, that its scores are saved under (lowercase letters, digits and _, different for each evaluator).
Kinds
| Kind | How it scores | Used in |
|---|---|---|
| Model judge | A model reads the reply with your rubric and gives pass or fail, a scale or a label | Experiments, live replies, Run evaluator, Try it |
| Code check | A fixed rule, such as "contains these phrases" or "is valid JSON" | Experiments, live replies, Run evaluator, Try it |
| Knowledge search | Ranks the agent's knowledge search against labelled passages | Agent experiments |
| Pairwise | A model reads two replies to the same question and picks the better one | Comparing two experiments |
Model judge templates
| Template | Checks | Scores | Needs |
|---|---|---|---|
| Correctness | The reply agrees with the reference answer | Pass or fail | A reference answer |
| Groundedness | The reply is supported by the knowledge the agent was given | Pass or fail | Knowledge ({{context}}) |
| Relevance | The reply answers the question asked | Pass or fail | – |
| Helpfulness | How useful the reply is | Scale 1 to 5 | – |
| Conciseness | No padding or repetition | Scale 1 to 5 | – |
| Safety | Nothing harmful, unsafe or inappropriate | Pass or fail | – |
| Tone of voice | The reply matches the tone you describe | Scale 1 to 5 | – |
| Custom rubric | Whatever your instructions say | Scale 1 to 5 | – |
- Instructions for the judge is the rubric. Insert adds
{{input}}(the question),{{output}}(the reply),{{reference}}(the reference answer) and{{context}}(the knowledge passages the agent was given). The question and the reply are added for you if you leave them out. - Choose how it scores: Pass or fail, a Scale (from 1 to 3, 4, 5, 7 or 10) or Labels (2 to 10, each with a score, such as Good 100%, Okay 50%, Bad 0%). A scale of n out of N is stored as (n − 1) / (N − 1), so 4 out of 5 is 0.75.
- A reply passes at 0.5 or more by default; choosing a scale sets the pass mark to 0.75. Change it under passes at.
- If a rubric uses
{{reference}}and an example has none, that reply scores n/a, and n/a is left out of averages.
Which model judges
Judge model is Same as the agent by default: each agent's own model and key grade its replies (for a workflow, the model of its first Agent block). Or pick one of your saved LLM keys and a model, for example a stronger model to grade a smaller one. Judges run at temperature 0 on your key: SentientOne charges no credits, and your provider bills the tokens. If a judge can't run (no key, a refused key, a timeout after 60 seconds), the score is empty with the reason as its comment, and the experiment carries on.
Code checks
| Check | Passes when | Notes |
|---|---|---|
| Contains | The reply includes every phrase | Up to 50 phrases, one per line; case ignored by default; partial credit for some |
| Doesn't contain | None of the phrases appear | Case ignored by default |
| Matches pattern | A regular expression matches | Case-sensitive unless you tick ignore case |
| Exact match | The reply equals the reference answer, whitespace collapsed | Case ignored by default; needs a reference answer |
| Valid JSON | The reply is JSON | A code fence around it is fine |
| Has JSON keys | The JSON has every key | Dotted paths such as order.items.0.sku; partial credit |
| Length | At most (or at least) a number of characters | 1,200 by default |
| Latency budget | The reply took less than a time limit | 5,000 ms by default; experiments only |
| Cost budget | The reply cost less than a limit | US$0.01 by default; experiments only |
Knowledge search and pairwise
- Recall@k, MRR, nDCG@k (k = 5 by default) and Rejection score an agent's knowledge search against passages labelled on each example. Examples can't be labelled with passages in the app yet, so these show n/a for now.
- Pairwise preference reads two replies to the same question and answers A, B or a tie. It's used on the compare page, not in experiments: see Compare experiments and promote a variant.
Try it
The editor's Try it panel scores a reply with the evaluator as it is now, before you save. Pick an Agent (and Use its latest reply, or type a Question and a Reply), add a Reference answer or Knowledge if the rubric uses them, and choose Try it. Nothing is saved and no credits are used; 20 tries a minute. Latency and cost budgets can't be tried here.
An evaluator's page
Tabs Settings, Live replies (score an agent's live replies: see Score live replies) and Recent scores (its last 50 scores and the experiments that used it). The kind can't change after you create it. An experiment can use up to 10 evaluators.
Still stuck?
Email us with what you tried and any request id. A person replies.