For teams shipping AI agents to real users
Recreate realistic conversations, tool calls and failures safely. Check what the agent said, what it did to your systems and what the customer sees—not just whether one answer looks right.
Early access: we are opening in batches. Request a spot and we will email you when you can start your 14-day trial, then plans from $299 a month.
How it works
Register your agent's staging endpoint. CertiQA calls it over HTTPS: OpenAI-compatible chat, Anthropic Messages or any REST webhook. The tools it calls are mocked, which means answered with responses you define, so nothing real is touched.
Write the conversation, the tool calls you expect and what must be true afterward, as steps on a visual canvas. An LLM judge (a model that grades answers) scores them against a rubric you write, a plain-English description of a good answer, and the case repeats because agents vary.
A shell script triggers the suite from any pipeline and exits on the quality gate. Every run produces a report, and a Release Report packages it for a reviewer.
What this looks like day to day
An agent’s answer is only one part of the outcome. Its tool choices, recovery from failures and effects on your product all matter.
An exact-match assertion passes or fails by luck. You need to know how often the agent is right, and who decides what right means.
A test must not refund a customer or drop a table, yet the bugs that matter live where the agent meets your own systems.
Prompt injection and jailbreaks are normal traffic for a public agent. A prompt tweak or model upgrade can quietly weaken a defense that held last week.
An agent can claim it completed a task while leaving the wrong data behind or showing the customer something different.
How CertiQA helps
Bring in production traces and turn real failures or edge cases into regression tests. Run them against your staging agent with controlled tools, so you can reproduce important behavior without replaying live side effects.
Production traces are inputs for testing; CertiQA is not a live production monitoring service.
An LLM judge scores each answer against a rubric you write, such as "grounded in the refund policy". Set a repeat count and a pass threshold and the case runs that many times, so the report shows a repeat pass rate with a 95% confidence interval instead of one lucky answer.
For anything exact, such as a tool argument or a variable, use a deterministic check with no judge involved.
A judge is a model and can be wrong. Each saved rubric carries a calibration summary against human-labeled examples so you can see how far to trust it.
CertiQA answers your agent's tool calls with the responses you define, so nothing real is touched. A Passthrough step lets one tool reach your real staging endpoint while every other tool stays mocked. A tool with neither a mock nor a route is never sent anywhere.
Then break things on purpose: inject a timeout, an HTTP 500 or 429, or malformed JSON on a tool, and grade how the agent recovers.
On hosted runners, private addresses are refused. To reach a system inside your network, use a self-hosted runner.
faults:
tool: fetch_customer_records
modes: [HTTP_429]
onCall: 1
expectRecovery:
inlineRubric: Tells the customer the
system is busy and never shows a
raw error body.
maxToolCalls: 3Run a sweep of 20+ built-in scenarios across seven categories of the OWASP LLM Top 10, the industry list of the main risks for AI applications, plus attacks generated from your agent's own tools, so a refund tool is tested for refund abuse and argument tampering. Each sweep is saved with a defense rate and a per-category breakdown, and a CI sweep can fail when the rate drops against the previous run that used the same attack set.
The catalog covers what an attacker can reach by talking to the agent. A clean sweep is a trend line for your team, not a security certification. Red-teaming is part of the trial and of Team Pro Plus and Enterprise Ultra.
How red-teaming works| Category | Defense rate | Defended / total |
|---|---|---|
PROMPT_INJECTION | 100% | 4 / 4 |
JAILBREAK | 100% | 3 / 3 |
SYSTEM_PROMPT_LEAK | 100% | 2 / 2 |
PII_DATA_LEAK | 67% | 2 / 3 |
TOOL_MISUSE_ESCALATION | 75% | 3 / 4 |
Every case is a list of steps on a visual canvas: what the user says, which tools are mocked, what must be true afterward. Your connected AI assistant can draft cases over MCP, and a lead, product owner or domain expert can read and approve them as steps, without opening generated code.
Browser tests run with CertiHeal, which repairs locators that drift when a page changes, and export plain Playwright TypeScript.
Reading steps shows what is checked, not that it is enough. It sits alongside code review.
AI assistants over MCPA single case can call an API, talk to an agent with mocked or real tools and check a page. See the example at the top of this page, or web UI automation.
Available on Team Pro and above.
Already using other tools?
Promptfoo and DeepEval evaluate prompts and outputs. LangSmith, Arize Phoenix and Braintrust trace and analyse what an agent did. Most agent teams want that, and CertiQA does not replace it.
CertiQA runs the whole journey before release: your agent's tool calls against mocked or staging systems, injected failures, the state left in your APIs and the page the customer sees, with a CI gate on the result.
CertiQA can take traces from Langfuse, Arize Phoenix or Helicone and turn real failures into regression tests.
Compare in detailPricing
CertiQA is in early access, so request a spot and we will email you when you can start. The trial has Team Pro Plus features and 500 credits, and needs no approval and no card. After it ends, access pauses until you subscribe. There is no free plan.
One developer or QA engineer testing their first AI agents.
$299/month
Teams that want agent tests to gate every release.
$749/month
Teams that also want red-teaming and cloud triage of CI failures.
$899/month
Organizations that need private deployment and custom terms.
Custom
One test across UI, API and agent is Team Pro and above. Red-teaming is Team Pro Plus and above.
How a test runs
A reply that says the seat is booked proves little. This is the example case from the top of the page, read as a journey.
An API Request step creates a test passenger in your staging system and keeps the booking token for later steps.
The public holiday calendar stays mocked. The booking tool reaches your staging API. The user asks for seat 12A and a rubric checks that the reply confirms the booking.
A second API Request fetches the booking and asserts the seat is 12A, whatever the agent said.
Browser steps open the booking page and assert it shows the same seat.
When it fails
Say the agent asks the booking tool for seat 14C but tells the customer it booked 12A. The rubric reads the reply, so that step passes. The API Request that reads the booking back is the step that fails, and the run report shows the failed assertion next to the agent's own tool call, so you can see where it went wrong.
An illustration of the example case, not a recording. The step names, transcript layout and failure wording follow what a CertiQA run report shows.
Agent transcript
USER Change my flight and book seat 12A.
TOOL reserve_flight { "seat": "14C" }
AGENT Done. Your flight is changed and seat 12A is booked.
Judge: PASSFailed step
HTTP API Verification Failed for GET /bookings/{id}:
- Assertion [JSON_PATH "$.seat" EQUALS 12A]: Expected "12A" but received "14C"When no API shows what the agent changed, a Database Query step can read the result of a query you have approved. It runs only on a self-hosted runner, and CertiQA never connects to your database.
Making it a gate
CertiQA runs the tests on its own runners. A plain shell script, which needs only curl and jq, triggers a suite from any CI system, waits for the run and exits on its quality gate.
Exit codes: 0 gate passed, 1 gate failed, 2 timed out waiting, 3 CertiQA API or authentication error.
export CERTIQA_PROJECT_ID="<project id>"
export CERTIQA_SUITE_ID="<suite id>"
export CERTIQA_TOKEN="<CI token>"
./certiqa-run.shWhen someone asks for proof
A Release Report collects what CertiQA measured for a project over a date range into one export, with a SHA-256 hash so a reviewer can tell if it was edited afterwards.
It is evidence for a human reviewer, not a certification. The audit-log section covers the whole platform, not just the project, and the pack says so.
FAQ
AI agent testing checks that an agent behaves correctly before it reaches customers: that it gives grounded answers, calls the right tools with the right arguments, recovers when a tool fails, and resists prompt injection and jailbreaks. Because an agent answers differently on every run, the checks are rubric-based and repeated, not a single exact-match assertion.
Read Agent Testing 101Register your agent's staging endpoint in CertiQA, mock the tools it calls so nothing real is touched, write the conversations and failures you expect as steps, and let an LLM judge score the answers against a rubric you write. Then run the suite from CI with a plain shell script. Its exit code tells your pipeline whether the quality gate passed.
See AI agent testingYou check behavior instead of reading every line. CertiQA cases are plain steps on a visual canvas: what the user says, which tools are mocked, what must be true afterward. Assistants such as Cursor and Claude Desktop can draft cases over MCP, and a person reads, edits and approves the steps. Web UI tests export to Playwright TypeScript when you want the code. Reviewing steps shows what is checked, not whether the checks are enough, so it complements code review rather than replacing it.
AI assistants over MCPRed-teaming means attacking your own agent before someone else does. CertiQA ships 20+ built-in scenarios across seven OWASP LLM Top 10 categories, including jailbreaks, instruction overrides, system-prompt extraction, data leakage and tool misuse. It also generates attacks from your agent's own tools, so a refund tool gets refund-abuse and argument-tampering attempts. Each sweep is saved as posture history, and a CI sweep can be set to fail when the defense rate drops against the previous run that used the same attack set. The catalog covers what a conversational attacker can reach; it is not a security certification.
How red-teaming worksCertiQA calls your agent over HTTPS: OpenAI-compatible chat endpoints, Anthropic Messages endpoints or any REST webhook. The tool calls your agent makes over REST or the Model Context Protocol (MCP) are intercepted and answered with the responses you define. For systems built with LangGraph, CrewAI or AutoGen, you can send OpenTelemetry GenAI spans and assert on which agent hands off to which. Using an MCP server directly as the agent under test is not available yet.
These platforms offer evaluation and observability across development and production. CertiQA focuses on safely exercising an agent end to end: control its tool responses, inject failures, verify resulting system state and customer-facing UI, and run the scenario as a CI gate. Most teams keep both: CertiQA can import traces from Langfuse, Arize Phoenix or Helicone and turn real failures into regression cases.
Compare CertiQAKeep them. Open-source libraries like these are a good fit for prompt and output evaluations that you run yourself. CertiQA covers the part around them: it mocks and routes the tools your agent calls, injects failures, checks the state left in your APIs and the page a customer sees, and gates CI. It is a hosted platform, so your team does not maintain that harness.
See the full comparisonYes. One test case can seed data through your REST API, let the agent call chosen tools against your staging systems, read the result back through the API, and check the page a customer would see. That confirms what the agent did to your systems, not only what it said it did. A request to a private address works only from a self-hosted runner. Hosted runners can reach public addresses.
Yes. You can record browser tests in Chrome, run them with CertiHeal, which repairs locators that drift when the page changes, and export plain Playwright TypeScript. Agent tests and UI tests run on the same platform and appear in the same release gate.
See web UI automationCertiQA calls your agent over HTTPS, so it needs a staging URL that CertiQA can reach. For agents inside a private network, Enterprise Ultra includes customer-hosted runners that sit inside your VPC and connect outbound over HTTPS, with no inbound firewall ports opened.
Security and deploymentCertiQA is in early access and is opening to teams in batches. Request a spot with a work email and we will email you when you can start. The 14-day trial itself needs no approval: it gives you Team Pro Plus features, including red-teaming, with 500 credits, and no credit card is needed. When the trial ends, access and CI runs pause until you choose a plan, so you only pay once it has proved useful on your own agent.
See pricingRegister a staging endpoint, mock one tool, write one rubric and run it. You will see what the report shows on your own agent before deciding anything else.
Talk to the team
We will look at how you test today and show you what CertiQA would add. Prefer email? Write to sales@certiqa.app.
If nothing opened, write to sales@certiqa.app and mention your email.