QA and AI agent testing glossary

A test tool behavior where an action or assertion waits until the element is ready, instead of the test sleeping for a fixed time. It is the main cure for timing-related flakiness.

A reusable, parameterized group of test steps, such as login or checkout, written once and used across many tests so a change is made in one place.

A range that shows how sure you can be about a measured pass rate. The Wilson score interval suits small samples: 10 passes out of 10 gives a 95% lower bound of only about 72%, which is why a single clean run is weak evidence.

The total spend on attempts divided by the number of attempts that succeeded, so failed retries count. A cheap model that fails often can cost more per success than a pricier one that rarely fails.

A selector that identifies an element on a web page for a test to click or check. Prefer stable hooks such as roles, labels or a test ID over position-based paths that break when the layout changes.

page.getByTestId("checkout-btn")

Giving an agent more tools, permissions or autonomy than its job needs, such as write access to a database when it only reads. It is an item on the OWASP Top 10 for LLM Applications.

Deliberately making a dependency misbehave in a test, for example by adding latency or returning a 429 or 500 error, to check that the system recovers sensibly.

Prepared data or state that a test needs before it runs, such as a user account or an order, created in a known, repeatable way.

A test that passes and fails on the same code. Common causes are timing races, shared state between tests and unstable test data. Flaky tests train teams to ignore failures.

Fix: wait on conditions instead of fixed sleeps, and isolate test data.

Whether the claims in an agent's answer are supported by the context it was given, such as retrieved documents or tool results. An ungrounded claim is a candidate hallucination.

When a model states something false or invented with confidence, such as a policy, a price or a function that does not exist.

A browser that runs without a visible window. It loads pages and runs scripts as usual, which makes it practical for automated tests on servers and in CI.

The property that doing something once or many times leaves the same result. Idempotent tests create unique data or clean up after themselves, so reruns do not collide.

const email = `qa+${Date.now()}@example.test`

An attack where hidden instructions sit in content the agent reads, such as a web page, a PDF or an email, rather than in the user's message. See also prompt injection.

A prompt crafted to make a model ignore its safety rules or role, often by role-play or by claiming special authority.

A model used to score another model's output against a written rubric, such as "grounded in the policy" or "does not reveal internal instructions". Its scores should be spot-checked against a person's.

An open protocol for connecting AI models and assistants to external tools and data sources through a common interface, so any compatible client can use any compatible server.

A stand-in for a real dependency that returns controlled responses. In agent testing, a mocked tool lets the agent act without touching a real system.

Producing different output for the same input. Language models can vary their wording, reasoning steps and tool order between runs, so tests check meaning, structure and state instead of exact text.

Trace data that follows the OpenTelemetry conventions for generative AI. In multi-agent systems, they show which agent called which model or tool, and in what order.

The set of configurations where you cannot improve one measure, such as accuracy, without worsening another, such as cost. Anything off the frontier is dominated by a better option.

Letting a tool call reach the real system, typically staging, instead of answering it with a mock. Used when a test needs the actual side effect or data.

Text that tries to override an agent's instructions. Direct injection comes from the user ("ignore all previous rules"). Indirect injection is hidden in content the agent reads. It is the language-model counterpart of SQL injection.

A common agent pattern in which the model reasons about the task, acts by calling a tool, observes the result and repeats until it can answer.

Testing a system by attacking it the way an adversary would. For agents it means prompt injection, jailbreaks, data extraction and tool misuse attempts, run before release and after every change.

A test that re-checks behavior that used to work, to catch a change that breaks it. Failed production conversations make good regression tests.

A dated export of what CertiQA measured for a project for a run or a date range: pass rates, failure recovery, attack resistance and judge calibration. A SHA-256 fingerprint makes later edits detectable. It is evidence for a reviewer, not a certification.

A hard limit on how many turns, tool calls or seconds an agent may use for one task. Exceeding it fails the test, which catches runaway loops early.

An agent revealing its hidden instructions when asked, for example to "repeat everything above". Instructions may contain business rules or secrets that should stay private.

Splitting a large suite across several machines that run in parallel, so total time falls roughly in proportion to the number of machines.

Catching an agent's tool call before it reaches a real system, checking its name and arguments against a contract, and returning a controlled response. It is how you test agents safely.

The definition of a tool's name and its arguments, usually JSON Schema. Tests validate the agent's arguments against it.

An assertion that retries until a condition holds or a timeout passes, instead of checking once. It removes most timing flakiness from UI tests.