AI Agent Pre-Flight Checklist: 20 Checks Before Release

Ticks are kept in this page only and reset when you reload. Not every item applies to every agent, so skip what does not.

1. Tool calls and isolation

Make sure an agent's actions cannot change live systems during automated runs.

1.1 All tool calls are intercepted. Every REST or MCP tool call is caught and handled in a controlled way before it reaches a real system.
1.2 No live data is changed by CI runs. Database writes, CRM updates and payment calls are mocked, or pointed at a disposable staging copy, in automated runs.
1.3 Arguments are validated. Every argument passed to a tool is checked against that tool's JSON Schema.
1.4 Failures are handled. The agent copes with slow responses, HTTP 429 rate limits and 500 errors without crashing, looping or inventing a result.

2. Limits and cost

Stop loops and delegation cycles before they become an incident or a bill.

2.1 A hard step limit is enforced. The agent is halted if it exceeds a set number of turns or tool calls for one request, and a test proves it.
2.2 Handoff cycles are caught. In multi-agent systems, traces show that agents never hand work back and forth indefinitely (A to B to A).
2.3 A cost cap exists per session. A single session that spends more than your chosen amount on model usage is stopped or handed to a person.
2.4 Latency has a bound. Response time, including time to first token for streaming replies, is measured against a limit you set.

3. Adversarial testing

Check that the agent resists prompt injection, jailbreaks and data leakage. The OWASP Top 10 for LLM Applications is a useful guide.

3.1 Direct override attempts fail. The agent refuses prompts such as "ignore previous instructions" or "you are now in developer mode".
3.2 The system prompt stays private. The agent does not reveal its instructions when asked to repeat or print what came before.
3.3 Indirect injection is handled. Hidden instructions in web pages, documents or emails the agent reads are not followed.
3.4 Data does not cross customers. Tests confirm the agent never exposes one customer's data in another customer's conversation.

4. Grounded answers

Check facts and policy with rubrics, and measure across repeated runs rather than one.

4.1 Answers are grounded. Factual claims in the reply are supported by the retrieved context or tool results, and unsupported claims are flagged.
4.2 Judge rubrics are specific. Judge prompts use narrow criteria and examples, and their scores are spot-checked against a person.
4.3 Important cases are repeated. Critical cases run several times so you see a pass rate with a confidence interval, not one lucky result.
4.4 Out-of-scope requests are refused. Questions outside the agent's remit get a polite refusal rather than a made-up answer.

5. Release gates and feedback

Turn test results into a decision that people can review and a loop that keeps improving.

5.1 The release record is traceable. Each sign-off records the tests, mock data and commit it covers, ideally with a fingerprint so later edits are detectable.
5.2 Pull requests are gated. CI fails when the pass rate drops below the threshold your team agreed.
5.3 Production failures become tests. Failed conversations from your observability tool are cleaned of personal data and added to the regression suite.
5.4 Secrets stay where policy requires. Mock data and credentials used in tests stay inside the environment your security policy allows.

How CertiQA helps. CertiQA can run many of these checks for you: tool mocking and failure injection, OWASP-based attack suites, repeated runs with pass-rate reporting, and a CI gate. See AI agent testing or pricing.

Frequently asked questions

What should I check before releasing an AI agent?

Check that its tool calls are intercepted and validated, that step and cost limits are enforced, that it resists prompt injection and does not leak data, that answers are grounded and scored over repeated runs, and that CI blocks a release when the pass rate drops. This checklist covers all five areas in twenty items.

Do I need to pass all twenty checks?

Not for every agent. A read-only agent that answers from documents needs fewer tool checks than one that issues refunds. Treat the list as prompts: mark an item as not applicable on purpose, and spend your effort on the checks that match what your agent can do.

How do I test that an agent resists prompt injection?

Write attacks as test cases: direct overrides, requests to reveal the system prompt, and hidden instructions inside documents or tool results. Run them against the agent on every model, prompt or tool change and assert that it refuses and that no forbidden tool call is made.

How many times should each agent test run?

Run important cases more than once, because one pass can be luck. Start with a handful of repeats and raise it for cases where a failure is costly. Judge the result by pass rate against a threshold your team agrees, not by a single run.

What is MCP security and why does it matter for testing?

The Model Context Protocol lets agents call external tools and data sources. Each connected server widens what the agent can do, so test that it only uses the tools it needs, that destructive tools require confirmation, and that content returned by a tool cannot redirect the agent.

Can I use this checklist without CertiQA?

Yes. The checks are general engineering practice and apply with any test setup. CertiQA is one way to run many of them: it mocks tool calls, injects failures, runs OWASP-based attacks and gates CI.

See how CertiQA tests agents

AI agent pre-flight checklist: 20 checks before release

1. Tool calls and isolation

2. Limits and cost

3. Adversarial testing

4. Grounded answers

5. Release gates and feedback

Frequently asked questions

Run these checks against your own agent.