Evaluation
Review queues
Send replies to people for review: rubrics, automatic rules for thumbs down and low scores, the reviewer screen with its keyboard shortcuts, and corrections that become dataset examples.
Guide2 min readUpdated 31 October 2026
A review queue is a list of replies waiting for a person: they score each one against a rubric and write what the agent should have said. Use it for the replies a machine can't judge well, and to turn bad replies into dataset examples.
Steps: create a queue
- 1
Open Review queues
Click Review queues under Evaluate and choose New queue.
- 2
Basics
A Name and Instructions for reviewers (up to 4,000 characters): what a good reply looks like.
- 3
Rubric
Up to 10 rows, each with a Label, a Key and a Type: Thumbs, Pass or fail, Scale (whole numbers from 0 to 10, 1 to 5 by default) or Labels (2 to 10, each with a score). It starts with Correct (pass or fail) and Helpfulness (1 to 5).
- 4
Dataset
Optionally pick a dataset: reviewers can add each reply to it, with their correction as the reference answer.
- 5
Send replies here automatically
Tick Thumbs down, and or Low scores (an evaluator's score below a percentage, 50% by default). Narrow it with From (any agent or one) and Sample (100% by default). Choose Create queue.
How replies arrive
- Send to review under a reply in Conversations, or on a reply in an experiment's results.
- The automatic rules, every 5 minutes, looking back 24 hours. A reply is never added twice.
- Thumbs down covers thumbs from the Playground, the widget and chat links, not scores sent with
POST /v1/feedback. Low scores watches one evaluator's scores.
Steps: review
- 1
Open the queue
Its Replies tab lists what's Waiting, Done or Skipped, and how each arrived. Choose Review to open the reviewer screen.
- 2
Read the reply
The screen shows the conversation up to the reply and the queue's instructions.
- 3
Score it
Answer each rubric row. Keys 1 to 9 answer the highlighted row, such as 1 for thumbs up or Pass and 2 for thumbs down or Fail.
- 4
Correct it
Add a Comment, and under Correction write what the agent should have said. Leave Add to … ticked to add the reply to the queue's dataset.
- 5
Submit
Press Enter or choose Submit & next; Skip moves on without scoring. At least one row needs a score.
You're done when
The queue shows nothing waiting, the scores show on the replies in Conversations, and corrected replies are examples in the dataset.
Each rubric row is saved as a score from a person under its key, replacing an earlier score with that key on the same reply. Scores show in Conversations, in the trace's Feedback tab and on the agent's Overview, and an alert can watch their average.
Still stuck?
Email us with what you tried and any request id. A person replies.