Test your agent before every change

This guide ties together generated runs, the Scenario Library and replays of Dataset calls. It covers four stages: before launch, building a regression suite, after every change, and turning production failures into tests. For the production side, see Best practices for agent monitoring.

​

Which tool for which question

​

Before launch

Simulation grades calls on your Pass/Fail evals, so set those up first. Pressure tests are generated from them, the Scenario Library needs at least one, and a replay without any only measures latency.

• Add Pass/Fail evals in Agent Settings > Evaluation Rules. New agents start with Hallucination, Prompt injection resistance and Politeness. Add the tool-call evals if your agent uses tools, and custom evals for your business rules. Score (1 to 5) evals don’t decide a call’s result.
• Set up your Intents, since Routine callers are generated from them. See Call classification.
• Create test profiles for the data your agent needs from a caller, such as a name, an account detail or an appointment date. Without a profile, the Simulated Caller has no caller-specific context.
• Run a generated simulation with a mix of Routine callers and Pressure tests. The default is 10 calls with 30% Pressure tests. Read the failures, fix your agent, and run again.
• Match your callers’ conditions. Set Language: to the languages and accents your callers use, and Environment: to the noise they call from (Café, Office, Street or Car, at Low, Medium or High). Each setting applies to the whole run, so run once per condition you care about.
• Narrow the targets with Narrow down › when you change one Intent or one eval, so the run spends its calls there.
• Upload recordings you already have (inbound agents). If you have real calls from another system, upload them to your Dataset and replay them before your agent takes a call in Tuner.

​

Build a regression suite

Your suite is two sets: Scenarios in the Scenario Library and calls in your Dataset. Add to both every time something fails.

• Save the Scenarios that caught a failure. On the Runs tab, expand the run, tick the calls and click Save to library. The call you saved from becomes the first entry in the Scenario’s Run history. See Save to library from a run.
• Write Custom Scenarios for edge cases you know about but haven’t seen in a run. For a Pressure Scenario, set a Target eval so it passes or fails on that eval alone. See Write a Custom Scenario.
• Tag your Scenarios (for example by Intent or by the eval they target) so you can find the right set for a change.
• Add real failing calls to your Dataset. On Simulations > Dataset, click Add call and pick the call from Production. Name it after the problem. See Capturing failures.
• Add golden calls: ordinary calls your agent handles well today. They show when a fix for one problem breaks something that used to work.

Saved Scenarios keep the situation, goal, type, Intent and Target eval. They don’t keep the language, background noise or test profile, so choose those on each rerun.

​

After every change

Rerun your suite after a prompt edit, a model swap, a new STT or TTS provider, or a deployment. Only one simulation run can be in progress per agent at a time, shared by generated runs, library runs and replays. A run whose calls have all ended doesn’t block the next one. When you compare two setups, change one thing at a time so any difference has one cause. See Compare two setups.

​

Turn production failures into tests

When monitoring finds a failure, such as a failed eval, a red flag or a failure outcome, add it to your suite before you fix it. Simulation calls don’t appear in the Production list when you add a Dataset call. To keep a failing simulated call, save its Scenario to the library.

​

Read the results

Each call shows one result. What to do depends on which one. See Run statuses for every status. The Simulated Caller improvises, so the same Scenario can pass once and fail once. Rerun a failing Scenario before you treat it as fixed or broken, and use a replay when you need the exact same caller.

​

Limits

​

Next steps