Score live replies

An evaluator can score your agent's real replies as they happen, not just a dataset. The scores show on each reply in Conversations and its trace, on the agent's Overview, and can drive review queues and alerts.

Steps

1. 1

   Open the evaluator

   Under Evaluate → Evaluators, open a model judge or a code check and go to Live replies.

2. 2

   Add an agent

   Under Add an agent, pick the Agent and the Sample: the share of replies to score, 1% to 100% (100% by default).

3. 3

   Pick the channels

   Widget, Chat link and API (Live) are ticked; tick Playground to score test chats too.

4. 4

   Start

   Tick Also score the last 24 hours to score recent replies straight away, then choose Score live replies.

How it works

• Every 5 minutes, each agent's new replies in the chosen channels are sampled and scored, up to 50 per agent each time. The sample is fixed per reply, so two evaluators at the same rate score the same replies.
• The judge sees the question the reply answered and the knowledge the agent was given, and the score replaces that evaluator's earlier score on the reply.
• Run now scores the new replies at once and tells you how many it scored and how many went to a review queue.
• Switch an agent off with On. When you switch it back on, it starts from that moment: replies from while it was off aren't scored.
• One evaluator can score each agent once. Model judges run on your LLM key, with no credits; code checks are free.