Skip to main content

​
What is a replay?

When you change your agent, you need to know what got better and what broke. A replay plays a call from your Dataset back to your agent, and Tuner scores it with your evals, so you can compare the result with the run before. Use replays to:
  • Reproduce a call that failed.
  • Prove that a fix worked.
  • Compare two setups, like a new model or provider, on the same calls.
The same Dataset call replayed three times before a fix, failing every time, and three times after it, passing every time

​
Dataset or scenario?

Tuner has two ways to test your agent again, and they are separate features. What differs is the caller.

Dataset

The caller is a recording of a real call, played back exactly as it was said. You replay it from the Replays tab.

Scenario

The caller is a simulated customer who follows a situation you saved. You run it from the Scenarios tab.

​
Replay a Dataset call

1

Pick the calls to replay

Go to Simulations > Replays and click New Replay. Next to Calls, click Change and tick up to 10 Dataset calls.
  • Calls that are still extracting, or that failed, can’t be picked. Add at least one Ready call to your Dataset first.
  • Replays run against inbound agents on Vapi, Retell, Dograh, LiveKit, Pipecat, or a Custom API integration. Set up inbound simulation first.
The New Replay dialog with three Dataset calls selected, each replayed three times
2

Run it

Tuner calls your agent and plays each caller turn from the recorded call.
  • Set Times to replay each call (1 to 20, at most 20 calls in total) and Max call duration (1 to 12 minutes). Longest possible call warns you if a call won’t fit.
  • Click Run. The run appears on the Replays tab and fills in as calls finish.
  • Tuner waits for your agent’s greeting, plays each caller turn in full, even if your agent talks over it, and waits for your agent’s reply before the next turn. After the last turn it waits for one more reply, then hangs up. A call also ends if your agent hangs up or it reaches the max call duration.
  • The caller never reacts to what your agent says. They say what they said in the original call. See The caller doesn’t adapt.
  • To change how long Tuner waits for your agent, see Turn-taking settings.
3

Read the results

Every replay is graded, so you can see what changed.
A replay run with nine calls: six passed and three failed
  • Each run card shows how many calls passed and failed, and the calls, repeats, and turn-taking settings it used.
  • Every call is graded against all of your agent’s Pass/Fail evals. A green check means none failed. A red X means at least one failed, or the call didn’t run. If your agent has no Pass/Fail evals, calls count as ran and the run measures latency only.
  • Click a row to open the call. A Replay call banner names the Dataset call it came from, and Replay details shows the source call and the exact settings used. The recording is stereo: caller on the left, agent on the right.
  • To compare two setups, keep the same calls, repeats, max call duration, and turn-taking settings, then compare the run cards and open matching calls. There’s no side-by-side view yet. See Benchmarking.
The Replay call banner on a call's details page

​
Turn-taking settings

This is optional. Expand Advanced turn-taking in New Replay to change how long Tuner waits for your agent.
Advanced turn-taking settings with the default values

​
Limits

Replays are also available through the public API, in the Simulation group of the API Reference.

​
Next steps

Introduction to Datasets

Save the calls you want to replay: failures, hard callers, and edge cases.

Scenarios

Build your own simulated callers, or save them from a run, and run them again.