Testing agents
Write test scenarios for your AI agent and run them with resolido test before every change goes live.
Changing a prompt, knowledge or a skill can fix one answer and break another. resolido test runs a set of customer scenarios against your agent and checks the answers – a quick regression test you can run locally or in CI.
npx resolido test init # creates resolido-tests/example.yaml
npx resolido test # runs every *.yaml in resolido-tests/✓ Shipping question (resolido-tests/shop.yaml, 9.8 s)
✗ Order status uses the skill (resolido-tests/shop.yaml, 14.1 s)
• tool "order" was not used
reply: Sorry, I can't look up orders. …
https://www.resolido.com/conversations/chat_k17…
✓ Angry customer is handed to a person (resolido-tests/shop.yaml, 7.2 s)
2 passed, 1 failed (14.1 s)Test files
agent: Support # agent id or name; --agent overrides it
tests:
- name: Shipping question
message: Do you ship to Slovakia and how much does it cost?
expect:
status: replied
contains: ["Slovakia"]
judge:
- States the shipping price to Slovakia
- Does not invent a delivery date
- name: Order status uses the skill
messages:
- Hi, where is my order?
- It's 1042
expect:
tools: ["order"]
not_contains: ["I don't know"]
- name: Angry customer is handed to a person
message: This is the third time I'm writing, I want to talk to a human NOW.
expect:
status: handed_offEach test starts a fresh test conversation. Several messages are sent one after another as one conversation; the checks apply to all AI replies.
| Expectation | Passes when |
|---|---|
status | The last turn ended as replied, handed_off (escalated to a person) or no_reply. |
tools / no_tools | A used tool's name contains / does not contain each text (case-insensitive). |
contains / not_contains | The replies contain / do not contain each text (case-insensitive). |
judge | An AI judge confirms each plain-language criterion against the conversation (up to 10). |
Options
| Flag | |
|---|---|
resolido test path/to/file.yaml dir/ | Run specific files or folders. |
--agent <id or name> | Run the tests against another agent, e.g. a copy you are editing. |
--filter <text> | Only tests whose file or name contains the text. |
--concurrency <n> | Tests in parallel, 1–5 (default 2). |
--cleanup | Delete the organization's test conversations afterwards. |
--json | Full results as JSON (default when not in a terminal). |
The command exits with 7 when a test fails, so CI jobs fail too:
- run: npx resolido test
env:
RESOLIDO_API_KEY: ${{ secrets.RESOLIDO_API_KEY }} # scopes: agents:read, testingGood to know
- Tests use AI credits like the dashboard playground – the agent's replies and the judge both count.
- Test conversations are hidden from the inbox and never trigger webhooks. Open the link printed next to a failed test to see the whole run, including tools and knowledge used.
- Live skills really run during tests. Point order-status tests at sample orders.
- Under the hood the CLI calls
POST /agents/{agentId}/testandPOST /test-conversations/{id}/judge, so you can build the same checks in any language.