Evals live in Agent Settings > Evaluation Rules, in the Evals section. They run on every production call and every Call Simulation call.
What are evals?
Evals are the criteria Tuner uses to judge whether your agent did the right thing on each call. Tuner has a library of predefined evals, such as Hallucination, Missed intent and Tool call missing. You can also define your own when you need something specific to your business:- Check for specific keywords or phrases: did the agent mention the daily special?
- Verify a specific action was taken: was an appointment booked?
- Monitor for compliance: did the agent read the required legal disclaimer?
Open the evals section
1
Go to Evaluation Rules
In Agent Settings, open the Evaluation Rules tab. The Evals section lists every eval active on this agent, both library evals and custom ones.
2
Click Add
Click Add in the top-right of the Evals section. The Add evals dialog opens, where you can pick from the library or start a custom eval.
Add evals from the library
The Add evals dialog groups the library by category: Accuracy, Actions & Tools, Conversation Quality and Safety & Compliance.
1
Browse or search
Use the category filters or the Search evals… box. Each card shows a short description and its scoring type, PASS/FAIL or SCORE 1-5.
2
Select and add
Select the evals you want, then click Add Eval (the button shows the count, for example Add 3 Evals). They appear in your Evals list and run on all future calls.
Create a custom eval
When the library doesn’t have what you need, click Create custom eval in the top-right of the Add evals dialog.
1
Set the Eval Label
Give your eval a clear, descriptive name, for example Professional Language or Disclaimer Read. This label appears in call results, dashboards and alerts.
2
Choose the Eval Type
Pick how the eval scores each call:
- Pass/Fail: binary evaluation. The call either meets the criteria or it doesn’t.
- Score 1-5: a scaled rating, for when you need a gradient (for example tone quality or thoroughness).
3
Write the Prompt Definition
Describe what should pass or fail in plain language. The LLM judge uses this as its rubric. Be specific: vague prompts produce inconsistent results.Example: “Check if the agent stated the following disclaimer verbatim: ‘This call may be recorded for quality assurance.’”Pass/Fail evals also get the call’s start time in UTC, so the prompt can check relative dates. Score 1-5 evals don’t. Example: “Pass if the agent booked the appointment for the day after the call.”For calls near midnight UTC, “tomorrow” can resolve to a different date than the caller’s local date.
4
Add Inputs Used
Under Inputs Used, choose which agent setup data the judge can reference when scoring the call. Open Add input from agent setup… and select one or more:
- System Prompt: the base instructions configured in Agent Setup
- Allowed Capabilities: what the agent is permitted to do
- Tools: the tools available to the agent
- Workflow: the agent’s workflow definition
- Metadata: a key from the per-call metadata you attach when you send calls (see Evaluate agents with per-call instructions)
5
Create the eval
Click Create Eval. The eval is added to your agent and runs on every future call, production and simulation alike.
Example: check for a disclaimer
To verify that your agent always reads a legal disclaimer, create a custom eval with:
If a call doesn’t meet this criterion, the eval fails. You can then create a red flag rule or an alert that fires when this eval fails.
Manage evals
From the Evals list on the Evaluation Rules tab, you can:- Edit an eval: click the pencil icon to change its label, prompt, type or inputs.
- Delete an eval: click the trash icon to remove it from the agent. It no longer runs on new calls. Deleting an eval pauses any rules or alerts that use it.
Limits
Limits enforced by the eval form in the dashboard:Next steps
Real-time alerts
Get notified when evals fail across your calls.
Diagnose your agent
Find which behaviors to measure next, based on real call data.