Evaluation

Test agents and workflows on datasets before you change them: evaluators that score replies, experiments that compare variants, scoring of live replies, review queues for people, and feedback from your users and your code.

1. 01How evaluation worksDatasets, evaluators, experiments, live scoring, review queues and feedback: what each one does, how they fit together, and what they cost.Concept2 min
2. 02Datasets and examplesCreate a dataset for an agent or a workflow, add examples by hand, from a CSV, from conversations, traces, experiments and review, or generate them, and how versions and splits work.Guide4 min
3. 03Evaluators: model judges and code checksThe evaluator templates (Correctness, Groundedness, Relevance and more), code checks, knowledge search metrics and pairwise judges: what each needs, how it scores, which model judges, and Try it.Reference4 min
4. 04Run an experimentRun an agent and its variants, or a workflow, over a dataset with your evaluators: repetitions, the 500-run limit, 1 credit per agent reply, live progress and reading the results.Guide3 min
5. 05Compare experiments and promote a variantSet a baseline, compare two to four experiments side by side, let a pairwise judge pick the better replies, and save the winning variant to the agent with Use it.Guide2 min
6. 06Score live repliesAttach an evaluator to an agent's live replies: sample rate, channels, the 5-minute schedule, Run now, and which evaluators can score live traffic.Guide2 min
7. 07Review queuesSend replies to people for review: rubrics, automatic rules for thumbs down and low scores, the reviewer screen with its keyboard shortcuts, and corrections that become dataset examples.Guide2 min
8. 08Feedback and scoresWhere scores come from (people, evaluators, end users, your code), the 0 to 1 scale, and how to send your users' ratings with POST /v1/feedback.Reference3 min