The task: FoodNow, the food delivery app we use as the example across our posts, is planning a lunch promotion. The business wants 60 orders a second for a minute, with orders answered in under 300 ms at the 95th percentile. We'll write the test most teams write first, watch it pass, and then find out why it shouldn't have.

Every run below is real, on one laptop, with:

The API under test (example code)

This is a small API we built for the guide, with two problems planted on purpose. Placing an order holds a lock for 20 ms, the way a row lock on the stock table would, so at most about 50 orders a second can get through. And when a dish sells out, it answers 200 OK with an error in the body.

# app.py: example API for this guide, not production code
import asyncio

from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()
MENU = [{"id": 1, "name": "Veg thali", "price": 17_900},
        {"id": 2, "name": "Butter naan", "price": 6_500}]
STOCK = {1: 2_000, 2: 100_000}       # portions left today
order_lock = asyncio.Lock()          # one order at a time, like a row lock on stock


class Order(BaseModel):
    user: str
    dish_id: int
    qty: int = 1


@app.get("/menu")
async def menu():
    return MENU


@app.post("/orders")
async def place_order(order: Order):
    async with order_lock:
        await asyncio.sleep(0.02)    # stock check + insert: 20 ms, so at most ~50 orders/s
        if STOCK[order.dish_id] < order.qty:
            return {"ok": False, "error": "sold out"}      # planted: 200 for a failure
        STOCK[order.dish_id] -= order.qty
        return {"ok": True, "order_id": sum(STOCK.values())}


@app.post("/reset")                  # test environments only
async def reset(thali: int = 2_000):
    STOCK.update({1: thali, 2: 100_000})
    return {"ok": True}

pip install fastapi uvicorn
uvicorn app:app --port 8020

k6 runs from Docker. --network host lets it reach the API on localhost:

docker run --rm --network host -v "$PWD:/w" -w /w grafana/k6 run test.js

Version 1: the test most teams write first

// v1.js
import http from "k6/http";
import { check, sleep } from "k6";

const BASE = __ENV.BASE || "http://localhost:8020";
const headers = { "Content-Type": "application/json" };

export const options = {
  vus: 20,
  duration: "60s",
  thresholds: {
    http_req_duration: ["p(95)<300"],
    http_req_failed: ["rate<0.01"],
  },
};

export default function () {
  for (let i = 0; i < 3; i++) http.get(`${BASE}/menu`);           // browse
  const res = http.post(`${BASE}/orders`,
    JSON.stringify({ user: `user${__VU}`, dish_id: 1 }), { headers });
  check(res, { "order placed": (r) => r.status === 200 });
  sleep(1);                                                         // think time
}

  █ THRESHOLDS
    http_req_duration
    ✓ 'p(95)<300' p(95)=22.68ms
    http_req_failed
    ✓ 'rate<0.01' rate=0.00%

    checks_succeeded...: 100.00% 1179 out of 1179
    http_reqs......................: 4716   77.299951/s
    iterations.....................: 1179   19.324988/s

Green. But look at the last line: 19 orders a second. The business asked for 60. This test never asked the question.

Step 1: Closed model vs open model (10 minutes)

The obvious fix is more users: 60 users, each placing about one order a second. Then there is the other way to describe the same load: start 60 orders every second, whatever happens. These two are not the same test, and k6 has a different executor for each:

// rush.js (imports, BASE and headers as in v1.js)
const closed = {                       // "60 users, each orders about once a second"
  executor: "constant-vus", vus: 60, duration: "60s",
};
const open = {                         // "60 orders start every second, whatever happens"
  executor: "constant-arrival-rate", rate: 60, timeUnit: "1s", duration: "60s",
  preAllocatedVUs: 100, maxVUs: 400,
};

export const options = {
  scenarios: { rush: __ENV.MODEL === "open" ? open : closed },
  thresholds: {
    http_req_duration: ["p(95)<300"],
    http_req_failed: ["rate<0.01"],
  },
};

export default function () {
  for (let i = 0; i < 3; i++) http.get(`${BASE}/menu`, { tags: { name: "menu" } });
  const res = http.post(`${BASE}/orders`,
    JSON.stringify({ user: `user${__VU}`, dish_id: 1 }),
    { headers, tags: { name: "orders" } });
  check(res, { "order placed": (r) => r.status === 200 });
  if (__ENV.MODEL !== "open") sleep(1);
}

The closed model (constant-vus), against the same API:

    ✓ 'p(95)<300' p(95)=277.53ms
    ✓ 'rate<0.01' rate=0.00%
    checks_succeeded...: 100.00% 2819 out of 2819
    iterations.....................: 2819   46.015457/s

The open model (constant-arrival-rate, run with -e MODEL=open):

level=warning msg="Insufficient VUs, reached 400 active VUs and cannot initialize more"
    ✗ 'p(95)<300' p(95)=7.75s
    ✓ 'rate<0.01' rate=0.00%
    checks_succeeded...: 100.00% 3202 out of 3202
    dropped_iterations.............: 398    5.810434/s
    iterations.....................: 3202   46.746256/s
level=error msg="thresholds on metrics 'http_req_duration' have been crossed"

Same API, same target, opposite verdicts. Here is why.

In the closed model, each virtual user waits for its answer before it starts the next order. When the API slows down, the users slow down with it, so the load drops to whatever the API can take. Here, 60 users only managed 46 orders a second, and the slowness never got the chance to build up. Real customers don't do that. A thousand people opening the app at 12:30 don't wait for each other's orders to finish.

In the open model, k6 starts 60 orders every second whatever happens, like real arrivals. The API can only finish about 47 a second, so the queue grows every second, and waiting times climb to 7.75 s. k6 ran out of virtual users to start new orders, and 398 orders never started at all. That is dropped_iterations.

Use the open model (constant-arrival-rate or ramping-arrival-rate) when the question is "can we handle N requests a second?" Use the closed model when you really have a fixed number of users who wait, such as 40 staff using an internal tool.

Step 2: Thresholds per endpoint, and on the rate itself (5 minutes)

The test above has one http_req_duration threshold for everything. Each order comes with three menu requests that take about 2 ms, so the slow requests are a small share of the total. Give each endpoint its own threshold by tagging requests with name (already done above) and adding a threshold on the tagged metric. While we're there, add a threshold on the thing the business actually asked for, the rate:

  thresholds: {
    http_req_duration: ["p(95)<300"],
    http_req_failed: ["rate<0.01"],
    "http_req_duration{name:orders}": ["p(95)<300"],
    "http_req_duration{name:menu}": ["p(95)<100"],
    iterations: ["rate>=58"],
  },

The closed model again:

    http_req_duration
    ✓ 'p(95)<300' p(95)=277.03ms
      {name:menu}
      ✓ 'p(95)<100' p(95)=1.68ms
      {name:orders}
      ✓ 'p(95)<300' p(95)=281.71ms
    http_req_failed
    ✓ 'rate<0.01' rate=0.00%
    iterations
    ✗ 'rate>=58' rate=46.116617/s
level=error msg="thresholds on metrics 'iterations' have been crossed"

To be fair to the blended number: here the orders on their own were also under 300 ms. Per-endpoint thresholds still earn their place. A 1.7 ms menu and a 282 ms order are different stories, and on an API where reads outnumber writes by more, the fast requests will hide the slow ones. But the threshold that caught this run is the rate. If the requirement is a rate, put the rate in a threshold. Otherwise a closed-model test can pass while delivering a fraction of the load.

Step 3: Checks don't fail a run (5 minutes)

Look again at the open-model run: checks_succeeded 100.00%, 3202 out of 3202. But there were only 2,000 thalis in stock. About 1,200 of those "orders" were the API saying "sold out", with status 200. The check only looked at the status.

Two problems, and both are common:

1. The check looked at the status code, not the answer. Check the body too.
2. A failing check doesn't fail the run. k6 reports checks, but the exit code only depends on thresholds. Add a threshold on checks.

And a third, about the test data: the test assumed there would be enough stock, without making sure. setup() runs once before the load starts; use it to put the data in a known state.

Here is the full version 3, open model, with all of that. It also reads its users from a CSV through SharedArray, so 400 virtual users share one copy instead of loading 400:

// rush_v3.js
import http from "k6/http";
import { check } from "k6";
import { SharedArray } from "k6/data";

const BASE = __ENV.BASE || "http://localhost:8020";
const headers = { "Content-Type": "application/json" };

// 500 test users, loaded once and shared by every VU.
const users = new SharedArray("users", () =>
  open("./users.csv").split("\n").slice(1).filter(Boolean));

export const options = {
  scenarios: {
    rush: {
      executor: "constant-arrival-rate", rate: 60, timeUnit: "1s", duration: "60s",
      preAllocatedVUs: 100, maxVUs: 400,
    },
  },
  thresholds: {
    "http_req_duration{name:menu}": ["p(95)<100"],
    "http_req_duration{name:orders}": ["p(95)<300"],
    http_req_failed: ["rate<0.01"],
    checks: ["rate>0.99"],
    dropped_iterations: ["count<1"],
  },
};

export function setup() {
  // Known data before every run: enough thalis for 60/s for 60 s.
  const res = http.post(`${BASE}/reset?thali=${__ENV.THALI || 5000}`);
  if (res.status !== 200) throw new Error(`reset failed: ${res.status}`);
}

export default function () {
  for (let i = 0; i < 3; i++) http.get(`${BASE}/menu`, { tags: { name: "menu" } });
  const user = users[Math.floor(Math.random() * users.length)];
  const res = http.post(`${BASE}/orders`,
    JSON.stringify({ user, dish_id: 1 }), { headers, tags: { name: "orders" } });
  check(res, {
    "order placed": (r) => r.status === 200 && r.json("ok") === true,
  });
}

export function handleSummary(data) {
  const m = data.metrics;
  const short = {
    orders_p95_ms: m["http_req_duration{name:orders}"].values["p(95)"],
    menu_p95_ms: m["http_req_duration{name:menu}"].values["p(95)"],
    orders_per_s: m.iterations.values.rate,
    dropped: m.dropped_iterations ? m.dropped_iterations.values.count : 0,
    checks_rate: m.checks.values.rate,
  };
  return {
    stdout: JSON.stringify(short, null, 2) + "\n",
    "summary.json": JSON.stringify(data, null, 2),
  };
}

handleSummary replaces k6's end-of-test output with your own. Here that is a short summary on screen and the full data in summary.json for CI to keep. One catch: once you return stdout, k6's usual summary is gone, so print what you need.

To see the check working, run it with the old stock of 2,000 thalis, -e THALI=2000:

{
  "orders_p95_ms": 8485.18389665,
  "menu_p95_ms": 1.63783975,
  "orders_per_s": 46.75651262857817,
  "dropped": 398,
  "checks_rate": 0.6246096189881324
}
level=error msg="thresholds on metrics 'checks, dropped_iterations, http_req_duration{name:orders}' have been crossed"

summary.json has the exact count: "passes": 2000, "fails": 1202. Every thali sold, then 1,202 failures that version 1 counted as successes. With the default 5,000 thalis the check passes, and the run still fails on speed and dropped orders. k6 exits with code 99 when a threshold is crossed, which turns a CI job red.

Step 4: Fix it and run again (5 minutes)

The fix on the API side, in the example: hold the lock only for the stock update, and do the slow part (here, an 18 ms payment call) outside it. And say "sold out" with a status that means it, 409 Conflict:

@app.post("/orders")
async def place_order(order: Order):
    async with order_lock:             # hold the lock only for the stock update
        await asyncio.sleep(0.002)     # 2 ms
        if STOCK[order.dish_id] < order.qty:
            raise HTTPException(409, "sold out")
        STOCK[order.dish_id] -= order.qty
        order_id = sum(STOCK.values())
    await asyncio.sleep(0.018)         # payment call: 18 ms, no lock needed
    return {"ok": True, "order_id": order_id}

Each order still takes about 20 ms, but only 2 ms of it is spent holding the lock. Version 3 again:

{
  "orders_p95_ms": 22.978024,
  "menu_p95_ms": 1.6793951,
  "orders_per_s": 59.98472014837337,
  "dropped": 0,
  "checks_rate": 1
}
exit 0

60 orders a second, orders answered in 23 ms, nothing dropped.

Step 5: Find the new limit (10 minutes)

Passing at 60 tells you nothing about 200. To find where the fixed API slows down, raise the arrival rate steadily and stop the test once it goes past the limit, with ramping-arrival-rate and abortOnFail:

// limit.js
import http from "k6/http";
import { check } from "k6";
import exec from "k6/execution";

const BASE = __ENV.BASE || "http://localhost:8020";
const headers = { "Content-Type": "application/json" };

export const options = {
  scenarios: {
    find_limit: {
      executor: "ramping-arrival-rate", startRate: 50, timeUnit: "1s",
      preAllocatedVUs: 200, maxVUs: 1000,
      stages: [{ target: 600, duration: "110s" }],   // +5 orders/s every second
    },
  },
  thresholds: {
    "http_req_duration{name:orders}": [
      { threshold: "p(95)<300", abortOnFail: true, delayAbortEval: "10s" },
    ],
  },
};

export function setup() { http.post(`${BASE}/reset?thali=1000000`); }

export default function () {
  const res = http.post(`${BASE}/orders`,
    JSON.stringify({ user: `user${exec.vu.idInTest}`, dish_id: 1 }),
    { headers, tags: { name: "orders" } });
  check(res, { "order placed": (r) => r.status === 200 });
}

    ✗ 'p(95)<300' p(95)=449.63ms
find_limit ✗ [  67% ] 0240/0256 VUs  1m14.0s/1m50s  419.85 iters/s
level=error msg="thresholds on metrics 'http_req_duration{name:orders}' were crossed;
  at least one has abortOnFail enabled, stopping test prematurely"

It stopped at 74 seconds, at about 420 orders a second. That is not the limit, and the reason is worth knowing: the threshold is a p95 over the whole run so far, including a minute of fast early requests, so it trips late. To see where it really bent, we ran it again with --out csv=limit.csv and worked out p95 for each 5-second slice:

The fixed API on this laptop holds to about 340 orders a second, then the queue builds quickly. That is more than five times the promotion's target. That's the number to write down, and to re-test whenever the order code changes.

Executors at a glance

Habits worth keeping

1. Requirement says "per second"? Use an arrival-rate executor, and put the rate in a threshold.
2. Watch dropped_iterations. Every dropped iteration is a request your test meant to send and didn't. Give it a threshold of count<1.
3. Tag requests by endpoint and set thresholds per tag.
4. Checks look at the answer, not just the status, and get a threshold. On their own they never fail a run.
5. Put the test data in a known state in setup(). "Sold out" halfway through is a data problem, not a performance result.
6. Don't read the limit off an aborted run. Look at the time series.

What's next

A laptop and one process are fine for finding a bug like this lock. To know what production can take, the load has to come from outside your network, at the right rate, and you need to compare each run with the last one. ProvaLab's load testing runs your k6 tests and JMeter plans on ProvaLab's own runners, compares each run with a baseline so a slower build can fail the pipeline, and keeps the trend across runs. See it in the two-minute product tour.

New to k6? Start with Your first load test with k6. For the same "is my test telling the truth?" questions applied to pytest, read pytest in practice.

This guide scans a small web app the way you'd scan a real one. We start with a scan that looks thorough and misses the main bug, then build a ZAP Automation Framework plan that signs in, stays signed in, finds the bug and fails the build. Then we write the tests for the bug no scanner will find. Every run below is real:

The app: FoodNow's order pages (example code)

FoodNow is the food delivery app we use as the example across our posts. This is a tiny version of its website with three bugs planted on purpose. Don't deploy it.

1. The order search page prints the search term back without escaping it: reflected cross-site scripting (XSS).
2. The session cookie is set without HttpOnly or SameSite.
3. The order page checks that you're signed in, but not that the order is yours.

# app.py: example app with planted vulnerabilities, never deploy
import html
import secrets

from fastapi import Cookie, FastAPI, Form, HTTPException
from fastapi.responses import HTMLResponse, RedirectResponse

app = FastAPI()
USERS = {"alice": "alice-pass-123", "bob": "bob-pass-456"}
ORDERS = {
    101: {"user": "alice", "items": "Veg thali x2", "address": "12 MG Road, Pune"},
    102: {"user": "bob", "items": "Butter naan x4", "address": "7 Park Street, Kolkata"},
}
SESSIONS: dict[str, str] = {}


def page(title: str, body: str) -> HTMLResponse:
    return HTMLResponse(f"<!doctype html><html lang='en'><head><title>{title}</title></head>"
                        f"<body><h1>{title}</h1>{body}</body></html>")


def current_user(session: str | None) -> str:
    user = SESSIONS.get(session or "")
    if not user:
        raise HTTPException(303, headers={"Location": "/login"})
    return user


@app.get("/", response_class=HTMLResponse)
def home():
    return page("FoodNow", "<p>Hot food, fast.</p><a href='/login'>Sign in</a>")


@app.get("/login", response_class=HTMLResponse)
def login_form():
    return page("Sign in", "<form method='post' action='/login'>"
                "<label>User <input name='username'></label>"
                "<label>Password <input name='password' type='password'></label>"
                "<button>Sign in</button></form>")


@app.post("/login")
def login(username: str = Form(...), password: str = Form(...)):
    if USERS.get(username) != password:
        return page("Sign in", "<p>Wrong user or password.</p><a href='/login'>Try again</a>")
    token = secrets.token_hex(16)
    SESSIONS[token] = username
    resp = RedirectResponse("/orders", status_code=303)
    resp.set_cookie("session", token)                      # planted: no HttpOnly, no SameSite
    return resp


@app.get("/orders", response_class=HTMLResponse)
def my_orders(session: str | None = Cookie(None)):
    user = current_user(session)
    rows = "".join(f"<li><a href='/orders/{oid}'>Order {oid}</a></li>"
                   for oid, o in ORDERS.items() if o["user"] == user)
    return page("Your orders", f"<p>Signed in as {html.escape(user)}. <a href='/logout'>Sign out</a></p>"
                "<form action='/orders/search'><input name='q'><button>Search</button></form>"
                f"<ul>{rows}</ul>")


@app.get("/orders/search", response_class=HTMLResponse)
def search(q: str = "", session: str | None = Cookie(None)):
    user = current_user(session)
    found = [oid for oid, o in ORDERS.items() if o["user"] == user and q.lower() in o["items"].lower()]
    return page("Search", f"<p>Results for {q}: {len(found)}</p>")   # planted: q not escaped


@app.get("/orders/{oid}", response_class=HTMLResponse)
def order(oid: int, session: str | None = Cookie(None)):
    current_user(session)                                   # planted: never checks the owner
    o = ORDERS.get(oid)
    if not o:
        raise HTTPException(404)
    return page(f"Order {oid}", f"<p>{html.escape(o['items'])}</p><p>{html.escape(o['address'])}</p>")


@app.get("/logout")
def logout(session: str | None = Cookie(None)):
    SESSIONS.pop(session or "", None)
    return RedirectResponse("/", status_code=303)

pip install fastapi uvicorn python-multipart
uvicorn app:app --host 0.0.0.0 --port 8030
docker pull zaproxy/zap-stable        # about 3.7 GB
mkdir wrk && chmod 777 wrk            # ZAP runs as a non-root user and writes reports here

Scan 1: the full scan, without signing in (2 minutes)

ZAP ships three ready-made scan scripts: zap-baseline.py (spider and passive checks only), zap-api-scan.py (from an OpenAPI file, as in the first guide) and zap-full-scan.py (spider plus active attacks). Full scan sounds thorough:

docker run --rm --network host -v "$PWD/wrk:/zap/wrk:rw" zaproxy/zap-stable \
  zap-full-scan.py -P 8099 -t http://localhost:8030/ -m 1 -r full-noauth.html -J full-noauth.json

(-P 8099 moves ZAP off port 8080, in case something on your machine already uses it. -m 1 gives the spider one minute.)

WARN-NEW: Missing Anti-clickjacking Header [10020] x 3
	http://localhost:8030/ (200 OK)
	http://localhost:8030/login (200 OK)
WARN-NEW: X-Content-Type-Options Header Missing [10021] x 3
WARN-NEW: Content Security Policy (CSP) Header Not Set [10038] x 3
WARN-NEW: Permissions Policy Header Not Set [10063] x 3
WARN-NEW: Absence of Anti-CSRF Tokens [10202] x 1
	http://localhost:8030/login (200 OK)
WARN-NEW: Cross-Origin-Resource-Policy Header Missing or Invalid [90004] x 9
FAIL-NEW: 0	FAIL-INPROG: 0	WARN-NEW: 6	WARN-INPROG: 0	INFO: 0	IGNORE: 0	PASS: 137

137 checks passed, 6 warnings, no failures, done in 1 minute 37 seconds. It looks like a clean bill of health. Look at the URLs instead: / and /login, nothing else. ZAP can't sign in, so it never saw the order pages, never got a session cookie, and never tried an attack on the search box. The XSS and the cookie are both still there.

Lesson one: before you read a scan's findings, read its list of URLs. A scan is only as good as the pages it reached.

Scan 2: an Automation Framework plan that signs in (10 minutes)

The packaged scripts are fine for a quick look. For anything with a login, use the Automation Framework: one YAML file that describes the app, how to sign in, and which jobs to run in which order. It goes in version control next to your code.

# wrk/zap.yaml
env:
  contexts:
    - name: foodnow
      urls:
        - http://localhost:8030/
      includePaths:
        - "http://localhost:8030.*"
      excludePaths:
        - "http://localhost:8030/logout"     # or the spider signs itself out
      authentication:
        method: form
        parameters:
          loginPageUrl: http://localhost:8030/login
          loginRequestUrl: http://localhost:8030/login
          loginRequestBody: "username={%username%}&password={%password%}"
        verification:
          method: poll
          pollUrl: http://localhost:8030/orders
          pollFrequency: 60
          pollUnits: requests
          loggedInRegex: "Signed in as"
          loggedOutRegex: "<h1>Sign in</h1>"
      sessionManagement:
        method: cookie
      users:
        - name: alice
          credentials:
            username: alice
            password: ${ALICE_PASSWORD}     # from the environment, never in the file
  parameters:
    failOnError: true
    progressToStdout: true
jobs:
  - type: spider
    parameters:
      context: foodnow
      user: alice
      maxDuration: 1
  - type: passiveScan-wait
  - type: activeScan
    parameters:
      context: foodnow
      user: alice
      maxScanDurationInMins: 5
  - type: report
    parameters:
      template: traditional-json
      reportDir: /zap/wrk
      reportFile: af-auth

What the important parts do:

• authentication tells ZAP how to sign in: which URL to post to, with {%username%} and {%password%} filled in from the user.
• verification is how ZAP knows it's still signed in. Every 60 requests it fetches /orders and looks for "Signed in as". Without this, a scan that gets signed out partway through carries on, testing the sign-in page over and over.
• excludePaths keeps the spider away from /logout. Forget it, and the spider clicks "Sign out" early on.
• ${ALICE_PASSWORD} comes from an environment variable, so the password lives in your CI secrets, not in the repository.

Run it:

docker run --rm --network host -e ALICE_PASSWORD=alice-pass-123 \
  -v "$PWD/wrk:/zap/wrk:rw" zaproxy/zap-stable \
  zap.sh -cmd -port 8099 -autorun /zap/wrk/zap.yaml

Job spider requesting URL http://localhost:8030/
Job spider found 6 URLs
Job activeScan finished, time taken: 00:01:10
Automation plan succeeded!

The sign-in worked: the report now has Cookie No HttpOnly Flag [10010], raised on the response to the /login post, which only a signed-in run sees. But there is no XSS. Look at the spider again: 6 URLs, starting from /. The home page links only to /login, and nothing the spider could follow from there led to /orders. ZAP signs in with its own login request; that doesn't tell the spider where the signed-in pages are. So it never found /orders, the search form, or /orders/search.

Lesson two: signing in isn't enough. The spider has to be able to reach the pages behind the login.

Scan 3: give the spider a way in (2 minutes)

Two small changes: add /orders to the context's URLs, and start the spider there.

      urls:
        - http://localhost:8030/
        - http://localhost:8030/orders
...
  - type: spider
    parameters:
      context: foodnow
      user: alice
      url: http://localhost:8030/orders
      maxDuration: 1

Job spider requesting URL http://localhost:8030/
Job spider requesting URL http://localhost:8030/orders
Job spider found 10 URLs
Job activeScan finished, time taken: 00:01:44

Now the report has what we planted:

[40012] High (Medium)  Cross Site Scripting (Reflected) x1
        GET /orders/search  param=q  attack=</p><scrIpt>alert(1);</scRipt><p>
[40026] High (High)    Cross Site Scripting (DOM Based) x2
        GET /orders/search?q=<script>alert(5397)</script>
        GET /orders#jaVasCript:/*-/*`/*\`/*'/*"/**/(/* */oNcliCk=alert(5397) )//...
[10010] Low (Medium)   Cookie No HttpOnly Flag x1

For a single-page app, you'd also add the spiderAjax job, which drives a real browser to find links that only exist after JavaScript runs. This app has no JavaScript, so the classic spider is enough.

Triage: three alerts, one bug (5 minutes)

Three High alerts. Before you file three tickets, read the evidence. The odd one is a DOM-based XSS on /orders, a page that has no JavaScript at all. ZAP's "Other info" for it lists the steps its browser took:

Access: http://localhost:8030/orders<PAYLOAD_0>
Write to /html/body/form/input the value: <PAYLOAD_0>
Click element: /html/body/form/button

It typed the attack into the search box and clicked Search, which loaded /orders/search?q=…. That's the same unescaped q that rule 40012 found. Three alerts, one bug, one fix. The DOM XSS rule reports where the browser started, not where the problem is. Scanners report symptoms; grouping them by cause is your job.

The other result worth a decision is Absence of Anti-CSRF Tokens on the sign-in form. Login CSRF is a real but small risk, so say we've decided to accept it for now and track it. Record that decision in the plan with an alertFilter job, placed before the scans so it is in place when the alert is raised:

jobs:
  - type: alertFilter
    alertFilters:
      - ruleId: 10202            # Absence of Anti-CSRF Tokens
        url: "http://localhost:8030/login"
        newRisk: "Info"          # decision: login CSRF accepted for now, see ticket
  - type: spider
  ...

Use filters for decisions like this, and for confirmed false positives (newRisk: "False Positive"), always with a comment saying why. A filter without a reason becomes a hole nobody remembers making.

Fail the build on High (2 minutes)

A scan that only writes a report never blocks anything. Add an exitStatus job at the end of the plan:

  - type: exitStatus
    parameters:
      errorLevel: High
      warnLevel: Medium

Against the buggy app:

Job exitStatus started
An alert has been raised with a risk of at least: Medium
An alert has been raised with a risk of at least: High
Automation plan failures:
	An alert has been raised with a risk of at least: High
Automation plan warnings:
	An alert has been raised with a risk of at least: Medium
exit 1

Exit code 1 fails the CI job. The login CSRF alert no longer counts towards it, because the filter turned it into Info.

What no ZAP run found (5 minutes)

Look back at the first screen: Alice could open /orders/102, Bob's order, with his address. None of the scans reported it, and none could have. To ZAP, /orders/102 returning a page is a success. It doesn't know which orders are Alice's. This is broken access control, the number one risk in the OWASP Top 10, and it is found by tests that know your business rules:

# tests/test_access.py
import os

import pytest
import requests

BASE = os.environ.get("BASE", "http://localhost:8030")


def signed_in(user, password):
    s = requests.Session()
    r = s.post(f"{BASE}/login", data={"username": user, "password": password})
    assert "Signed in as" in r.text
    return s


@pytest.fixture
def alice():
    return signed_in("alice", "alice-pass-123")


def test_alice_can_read_her_own_order(alice):
    r = alice.get(f"{BASE}/orders/101")
    assert r.status_code == 200
    assert "12 MG Road" in r.text


def test_alice_cannot_read_bobs_order(alice):
    r = alice.get(f"{BASE}/orders/102")
    assert r.status_code == 404          # 404, not 403: don't confirm the order exists
    assert "Park Street" not in r.text


def test_signed_out_user_is_sent_to_login():
    r = requests.get(f"{BASE}/orders/101", allow_redirects=False)
    assert r.status_code == 303
    assert r.headers["location"] == "/login"

$ pytest -q tests
E       assert 200 == 404
E        +  where 200 = <Response [200]>.status_code
1 failed, 2 passed in 0.16s

Fix everything and run both again (5 minutes)

 app = FastAPI()
+
+
+@app.middleware("http")
+async def security_headers(request, call_next):
+    resp = await call_next(request)
+    resp.headers["Content-Security-Policy"] = "default-src 'self'; frame-ancestors 'none'; form-action 'self'"
+    resp.headers["X-Frame-Options"] = "DENY"
+    resp.headers["X-Content-Type-Options"] = "nosniff"
+    return resp

-    resp.set_cookie("session", token)
+    resp.set_cookie("session", token, httponly=True, samesite="lax")  # add secure=True behind HTTPS

-    return page("Search", f"<p>Results for {q}: {len(found)}</p>")
+    return page("Search", f"<p>Results for {html.escape(q)}: {len(found)}</p>")

-    current_user(session)
+    user = current_user(session)
     o = ORDERS.get(oid)
-    if not o:
+    if not o or o["user"] != user:                          # someone else's order: same as missing

$ pytest -q tests
3 passed in 0.09s

$ docker run ... zap.sh -cmd -port 8099 -autorun /zap/wrk/zap.yaml
Job spider found 10 URLs
Job exitStatus started
Job exitStatus finished, time taken: 00:00:00
Automation plan succeeded!
exit 0

What is left in the report is Informational only: the sign-in request and session cookie ZAP identified, and the login CSRF we decided to accept. The spider still found 10 URLs, so the clean result means "tested and clean", not "never reached".

In CI

The same command runs in any pipeline that has Docker. Start the app, run the plan with the password from your secrets, and keep the report:

      - name: ZAP scan (signed in)
        run: |
          docker run --rm --network host \
            -e ALICE_PASSWORD="${{ secrets.ZAP_TEST_PASSWORD }}" \
            -v "$PWD/wrk:/zap/wrk:rw" zaproxy/zap-stable \
            zap.sh -cmd -port 8099 -autorun /zap/wrk/zap.yaml
      - uses: actions/upload-artifact@v4
        if: always()
        with: { name: zap-report, path: wrk/af-auth.* }

Scan a test environment with test accounts, never production, and never with a real customer's login. An active scan sends thousands of attack requests, and fills in every form it finds.

Habits worth keeping

1. Read the list of URLs before the findings. A clean scan of two pages proves nothing about the rest.
2. Use an Automation Framework plan for anything with a login, kept in the repo, with credentials from the environment.
3. Verify the session with a logged-in check, and exclude logout.
4. Give the spider start URLs behind the login. Logging in doesn't tell it where the pages are.
5. Group alerts by cause before filing them. Three alerts here were one line of code.
6. Record every filter's reason in the plan.
7. Fail the build with exitStatus, and pair the scan with access-control tests. The scanner finds the generic mistakes; the tests find the ones that depend on who owns what.

What's next

A scanner finds the generic mistakes in a running app. To cover the rest, you want code scanning, dependency checks and secrets detection on every change, and the findings kept in one place where someone triages them. ProvaLab AppSec runs code, dependency, secrets and running-app scans on your projects, and keeps each finding's history so a fixed issue can be re-checked rather than re-filed. See it in the two-minute product tour.

New to ZAP? Start with Your first security scan with OWASP ZAP. The other tool guides in this run: pytest in practice and k6 in practice.

Cross-browser testing runs the same tests on the different browser engines: Chromium (Chrome, Edge, Opera, Samsung Internet), Gecko (Firefox) and WebKit (Safari). Playwright can drive all three from one test.

This post does that for a FoodNow page with two planted bugs. Before planting them, we asked the three engines a set of questions to find out what really differs between them in late 2026, rather than repeating old advice. Every result below is from a real run, in Microsoft's official Playwright Docker image, version 1.64.0, with Chromium 156, Firefox 157 and WebKit 27.2.

Why the Docker image

On a normal Linux laptop, Chromium and Firefox run once Playwright has downloaded them. WebKit didn't: it needs system libraries that have to be installed with sudo. Playwright's error said exactly which:

║ Host system is missing dependencies to run browsers. ║
║     sudo playwright install-deps                     ║
║     sudo apt-get install libevent-2.1-7t64\          ║
║         libgstreamer-plugins-bad1.0-0\               ║
║         libflite1\                                   ║
...

The official image has all three browsers and everything they need, which is also how most teams run them in CI:

docker pull mcr.microsoft.com/playwright/python:v1.64.0-noble
docker run --rm --ipc=host -v "$PWD":/work -w /work mcr.microsoft.com/playwright/python:v1.64.0-noble \
  bash -c "pip install -q playwright==1.64.0 pytest==9.2.0 && python -m pytest -v"

Two details: the image contains the browsers but not the Python package, so install the same Playwright version inside it. And --ipc=host gives Chromium enough shared memory, which avoids crashes on bigger pages.

WebKit in Playwright is the WebKit engine on Linux, not Safari on a Mac. It catches engine differences, which are most of them, but not everything that's specific to macOS or iOS. For that, you need real Apple devices, as in Part 6.

Step 1: Ask the engines what differs

This script asks each engine the same questions, all in the same browser locale (en-IN) and time zone (India):

"""Ask each browser engine the same questions and print the answers side by side."""
from playwright.sync_api import sync_playwright

CHECKS = {
    "Date('2026-11-12 19:30')": "String(new Date('2026-11-12 19:30'))",
    "Date('2026-11-12T19:30')": "String(new Date('2026-11-12T19:30').getHours())",
    "Date('12 Nov 2026 19:30')": "String(new Date('12 Nov 2026 19:30').getHours())",
    "en-IN time": "new Date(2026,10,12,19,30).toLocaleTimeString('en-IN',{timeStyle:'short'})",
    "en-IN money": "new Intl.NumberFormat('en-IN',{style:'currency',currency:'INR'}).format(125000.5)",
    "button height": "(()=>{const b=document.createElement('button');b.textContent='Pay';document.body.append(b);return b.getBoundingClientRect().height})()",
    "input height": "(()=>{const b=document.createElement('input');document.body.append(b);return b.getBoundingClientRect().height})()",
    "date input type": "(()=>{const i=document.createElement('input');i.type='date';return i.type})()",
    "scrollbar width": "(()=>{const d=document.createElement('div');d.style.cssText='width:100px;height:100px;overflow:scroll';document.body.append(d);return 100-d.clientWidth})()",
    "user agent": "navigator.userAgent.slice(0,90)",
}
with sync_playwright() as p:
    for bt in (p.chromium, p.firefox, p.webkit):
        b = bt.launch()
        page = b.new_page(locale="en-IN", timezone_id="Asia/Kolkata")
        page.set_content("<!doctype html><body></body>")
        print(f"== {bt.name} {b.version}")
        for name, js in CHECKS.items():
            try:
                print(f"  {name:<28} {page.evaluate(js)!r}")
            except Exception as e:
                print(f"  {name:<28} ERROR {str(e).splitlines()[0][:80]}")
        b.close()

The results, side by side:

The date row matters because of old advice. For years, new Date("2026-11-12 19:30"), with a space instead of a T, returned "Invalid Date" in Safari, and countless bugs came from it. In WebKit 27.2 it parses like the others. The JavaScript standard still leaves non-standard date strings up to each engine, so prefer 2026-11-12T19:30 or separate numbers. But the specific trap many testers still check for didn't appear here.

What did differ: the size of form controls. A button is 21 pixels tall in Chromium, 23 in Firefox and 30 in WebKit, with the same CSS. And Firefox measured 25 pixels on our laptop but 23 in the Docker image, because the fonts differ. Same browser, different machine, different layout.

We also checked newer features:

CHECKS = {
    "Temporal": "typeof Temporal",
    "Intl.DurationFormat": "typeof Intl.DurationFormat",
    "CSS field-sizing": "CSS.supports('field-sizing','content')",
    "CSS anchor-name": "CSS.supports('anchor-name','--a')",
    "View transitions": "typeof document.startViewTransition",
    "Uint8Array.fromBase64": "typeof Uint8Array.fromBase64",
    "RegExp.escape": "typeof RegExp.escape",
    "Error.isError": "typeof Error.isError",
    "showPicker": "typeof HTMLInputElement.prototype.showPicker",
    "Element.moveBefore": "typeof Element.prototype.moveBefore",
    "CSS text-wrap: pretty": "CSS.supports('text-wrap','pretty')",
    "Navigation API": "typeof navigation",
}

Ten of the twelve were supported in all three engines. Two weren't:

text-wrap: pretty only changes how lines break, so a browser without it just ignores it. moveBefore() is a JavaScript method: calling it where it doesn't exist throws an error, and whatever it was doing doesn't happen.

Step 2: The page

A FoodNow "order for later" page with three parts: the delivery time, a list of dishes with a "Put my usual first" button, and a promo code row:

<style>
  /* The promo row was sized to fit the button exactly as it looked in Chrome. */
  .promo { display: flex; gap: 8px; height: 21px; overflow: hidden; }
  .promo input { flex: 1; }
</style>
...
<p>Your lunch arrives at <span id="slot"></span>.</p>
...
<ul id="dishes"><li>Masala chai</li><li>Veg biryani</li><li id="usual">Paneer wrap</li></ul>
<button id="usual-first">Put my usual first</button>
...
<div class="promo"><input id="code" placeholder="Promo code"><button id="apply">Apply</button></div>
<script>
  // The server sends the slot as "YYYY-MM-DD HH:MM", in the kitchen's time.
  const slotFromServer = "2026-11-12 19:30";
  const when = new Date(slotFromServer);
  document.getElementById("slot").textContent =
    when.toLocaleTimeString("en-IN", { hour: "numeric", minute: "2-digit" });

  // Move the customer's usual dish to the top, keeping its state (moveBefore doesn't reset it).
  document.getElementById("usual-first").onclick = () => {
    const list = document.getElementById("dishes");
    list.moveBefore(document.getElementById("usual"), list.firstElementChild);
  };
</script>

Two planted bugs, both built on assumptions that are true in Chrome: a row exactly as tall as Chrome's button, and a DOM method that Chrome has. The date string with a space is there on purpose too, to check the old Safari trap in a real page.

Step 3: One test, three browsers

A parametrized fixture runs every test once per engine. tests/conftest.py:

import functools
import http.server
import threading
from pathlib import Path

import pytest
from playwright.sync_api import sync_playwright

SITE = Path(__file__).parents[1] / "site"


@pytest.fixture(scope="session")
def base_url():
    handler = functools.partial(http.server.SimpleHTTPRequestHandler, directory=SITE)
    server = http.server.ThreadingHTTPServer(("127.0.0.1", 0), handler)
    threading.Thread(target=server.serve_forever, daemon=True).start()
    yield f"http://127.0.0.1:{server.server_port}"
    server.shutdown()


@pytest.fixture(scope="session")
def playwright():
    with sync_playwright() as p:
        yield p


@pytest.fixture(params=["chromium", "firefox", "webkit"])
def page(request, playwright, base_url):
    browser = getattr(playwright, request.param).launch()
    page = browser.new_page(locale="en-IN", timezone_id="Asia/Kolkata", viewport={"width": 412, "height": 500})
    page.goto(base_url + "/index.html")
    yield page
    page.screenshot(path=f"shots/{request.node.name}.png")
    browser.close()

(If you use the pytest-playwright plugin, pytest --browser chromium --browser firefox --browser webkit does the same.)

The tests, in tests/test_xbrowser.py:

def test_slot_time_is_shown(page):
    assert page.text_content("#slot") == "7:30 pm"


def test_apply_button_is_not_cut_off(page):
    # The button's box can be squeezed to fit the row, so check that its *content* fits in the box.
    content, inside = page.evaluate(
        "() => { const b = document.getElementById('apply'); return [b.scrollHeight, b.clientHeight]; }")
    assert content <= inside + 1, f"the button's text needs {content}px but has {inside}px"


def test_usual_dish_moves_to_the_top(page):
    errors = []
    page.on("pageerror", lambda e: errors.append(str(e)))
    page.click("#usual-first")
    assert errors == []
    assert page.locator("#dishes li").all_inner_texts()[0] == "Paneer wrap"

page.on("pageerror", ...) collects JavaScript errors that the page throws. Without it, a broken click often just "does nothing", and the test fails later with a confusing message, or not at all.

Step 4: Run it

tests/test_xbrowser.py::test_slot_time_is_shown[chromium] PASSED         [ 11%]
tests/test_xbrowser.py::test_slot_time_is_shown[firefox] PASSED          [ 22%]
tests/test_xbrowser.py::test_slot_time_is_shown[webkit] PASSED           [ 33%]
tests/test_xbrowser.py::test_apply_button_is_not_cut_off[chromium] PASSED [ 44%]
tests/test_xbrowser.py::test_apply_button_is_not_cut_off[firefox] PASSED [ 55%]
tests/test_xbrowser.py::test_apply_button_is_not_cut_off[webkit] FAILED  [ 66%]
tests/test_xbrowser.py::test_usual_dish_moves_to_the_top[chromium] PASSED [ 77%]
tests/test_xbrowser.py::test_usual_dish_moves_to_the_top[firefox] PASSED [ 88%]
tests/test_xbrowser.py::test_usual_dish_moves_to_the_top[webkit] FAILED  [100%]
E   AssertionError: the button's text needs 23px but has 17px
E   assert ['list.moveBe...s undefined)'] == []
========================= 2 failed, 7 passed in 11.27s =========================

Both bugs, only in WebKit, and the date string is fine in all three.

The reorder button does nothing in WebKit. The full error:

list.moveBefore is not a function. (In 'list.moveBefore(document.getElementById("usual"), list.firstElementChild)', 'list.moveBefore' is undefined)

An iPhone user taps "Put my usual first", and nothing happens. No message, nothing in the server logs.

The Apply button is cut off in WebKit. WebKit's button needs 23 pixels for its content. The row gives it 21, of which 17 are inside the border.

Our first version of this test passed in all three browsers. It compared the button's box with the row's box, and both were 21 pixels: in a flex row, the browser squeezes the button's box to fit the row, and the content spills out of the box and is cut off. Only the screenshots showed the problem. Firefox's content is 1 pixel taller than its box (18 against 17), which is rounding you can't see, so the check allows 1 pixel. Compare what users see, and look at the screenshots of every engine at least once.

Step 5: Fix it

-  .promo { display: flex; gap: 8px; height: 21px; overflow: hidden; }
+  .promo { display: flex; gap: 8px; align-items: center; }
...
-    list.moveBefore(document.getElementById("usual"), list.firstElementChild);
+    const usual = document.getElementById("usual");
+    if (list.moveBefore) list.moveBefore(usual, list.firstElementChild);
+    else list.insertBefore(usual, list.firstElementChild);  // Safari: no moveBefore yet

• No fixed heights for rows with form controls. Let the content decide, and use align-items: center to line things up.
• Check before you use a newer API. If the method exists, use it. If not, fall back to an older one that does the same job. insertBefore has been everywhere for decades. It doesn't keep some element state that moveBefore keeps, which doesn't matter for a plain list.

Three runs on the fixed page:

9 passed in 11.05s
9 passed in 11.28s
9 passed in 11.05s

What to run on which browsers

Running every test on every engine triples the time. Most teams don't need to:

• Run everything on one engine, usually Chromium, on every commit.
• Run a cross-browser set on all three engines, on every pull request or every night: layout-heavy pages, forms, anything using newer APIs, and the main customer journeys.
• Check your own analytics. If 40% of your customers use iPhones, WebKit isn't optional.
• Visual baselines per engine. Fonts and controls differ, so a screenshot from Firefox won't match one from Chromium. Keep one baseline per engine, as in Part 3.
• Real devices for the last mile. Playwright's WebKit catches engine differences. Real iPhones catch the rest.

Common mistakes

• Testing only in Chrome. iPhone browsers almost all run on WebKit. A Chrome-only test suite says nothing about them.
• Repeating old browser lore without checking. The famous Safari date-parsing trap didn't happen in WebKit 27.2. Ask the engines, as in Step 1.
• Sizing layouts to one engine's controls. Default buttons were 21, 23 and 30 pixels tall in our run.
• Checking boxes instead of content. Our first check passed while WebKit cut the button off.
• Using new APIs without a check. moveBefore exists in two engines out of three.
• Ignoring page errors. Listen for pageerror, so a broken click fails with the real reason.
• Different machines, same baseline. Fonts differ between a laptop and a container, so layouts do too. Run visual tests in the same image every time.

The end of Testing in Practice, for now

This is the last part of the series. Eighteen hands-on posts, each built on FoodNow, each run for real:

1. Accessibility with axe and Playwright
2. API testing with Postman and Newman
3. Visual regression testing
4. Testing database changes
5. Test doubles: mocks, stubs and fakes
6. Mobile app testing with Appium
7. Contract testing with Pact
8. BDD with pytest-bdd
9. User acceptance testing
10. Usability testing
11. Your first Selenium test
12. Your first Cypress test
13. Test estimation
14. The test summary report
15. Localization testing
16. Flaky tests
17. Property-based testing with Hypothesis
18. Cross-browser testing (this post)

One topic is still to come: running these suites automatically in CI with GitHub Actions, so a red test blocks a merge. We'll add it when it's ready.

ProvaLab's QA Infrastructure runs Playwright and Selenium tests on browsers in the cloud, with a live view of the browser while each test runs, and ProvaLab's plugins send each result to the ProvaLab test management system with its screenshots, so a parametrized run like this one shows which test failed on which browser. See it in the two-minute product tour.

One flaky test is an annoyance. Twenty of them change how a team behaves. People stop reading red builds, because "it's probably flaky". They press re-run until it's green. And then a real failure arrives and gets re-run away like all the others.

This series has already met two flaky tests by accident:

• In Part 12, a Cypress test passed 2 runs in 8 on a page with a real race bug.
• In Part 14, a test failed every night from 21:46 to 07:59 because it read the real clock.

This post takes flakiness on deliberately. We wrote a small FoodNow module with five tests and four classic causes of flakiness, measured how often each test fails over 100 runs, reproduced the failures, tried the usual shortcut (retries), and fixed each cause. Every number below is from a real run.

The code and its tests

foodnow/basket.py:

"""FoodNow basket and pricing."""
import random
import threading
import time
from datetime import datetime

PRICES = {"Paneer wrap": 180, "Masala chai": 40, "Veg biryani": 260}

_basket: list[str] = []  # one basket for the whole process


def add(dish: str) -> None:
    _basket.append(dish)


def items() -> list[str]:
    return list(_basket)


def total() -> int:
    """Food total in rupees. Orders of 10 or more dishes get 10% off."""
    food = sum(PRICES[d] for d in _basket)
    if len(_basket) > 10:
        food = food * 90 // 100
    return food


def is_kitchen_open(now: datetime | None = None) -> bool:
    now = now or datetime.now()
    return 11 <= now.hour < 23


class EtaService:
    """Works out the delivery time in the background, like a call to a maps service."""

    def __init__(self):
        self.minutes: int | None = None

    def start(self) -> None:
        def work():
            time.sleep(random.uniform(0.05, 0.15))  # the maps service takes 50-150 ms
            self.minutes = 25
        threading.Thread(target=work, daemon=True).start()

And tests/test_basket.py. Each test looks reasonable on its own:

import random
import time

from foodnow import basket


def test_new_basket_is_empty():
    assert basket.items() == []
    assert basket.total() == 0


def test_add_a_wrap():
    basket.add("Paneer wrap")
    assert basket.total() == 180


def test_kitchen_is_open_now():
    assert basket.is_kitchen_open()


def test_delivery_estimate():
    eta = basket.EtaService()
    eta.start()
    time.sleep(0.1)  # give the maps service time to answer
    assert eta.minutes == 25


def test_big_orders_get_ten_percent_off():
    # Any order of 10 to 20 wraps gets 10% off.
    n = random.randint(10, 20)
    basket._basket.clear()
    for _ in range(n):
        basket.add("Paneer wrap")
    assert basket.total() == 180 * n * 90 // 100

Install pytest and three plugins that help with flaky tests:

pip install pytest==9.2.0 pytest-randomly==5.1.0 pytest-repeat==0.9.4 pytest-rerunfailures==16.7

• pytest-randomly runs the tests in a different random order every time, and prints the seed so you can repeat an order.
• pytest-repeat runs a test many times (--count=50).
• pytest-rerunfailures re-runs failed tests (--reruns 2). We use it in Step 4 to show why it isn't the answer.

Step 1: Run it once

In the order the tests are written, with random ordering turned off:

$ pytest -p no:randomly -q
FAILED tests/test_basket.py::test_delivery_estimate - assert None == 25
1 failed, 4 passed in 0.19s

One failure, which looks like one problem. A single run tells you almost nothing about flakiness. You have to run the suite many times, and in different orders, and count.

Step 2: Measure it

This script runs the whole suite 100 times, each time in a new random order, and counts how often each test fails. Save it as flake_rate.py:

"""Run the test suite many times, each in a new random order, and report how often each test fails."""
import collections
import subprocess
import sys
import tempfile
import xml.etree.ElementTree as ET
from pathlib import Path

runs = int(sys.argv[1]) if len(sys.argv) > 1 else 100
extra = sys.argv[2:]
fails, seen = collections.Counter(), collections.Counter()
with tempfile.TemporaryDirectory() as tmp:
    for i in range(runs):
        report = Path(tmp) / f"run{i}.xml"
        subprocess.run([sys.executable, "-m", "pytest", "-q", "-p", "randomly", f"--junitxml={report}", *extra],
                       capture_output=True)
        for tc in ET.parse(report).iter("testcase"):
            name = tc.get("name")
            seen[name] += 1
            if tc.find("failure") is not None or tc.find("error") is not None:
                fails[name] += 1

print(f"{runs} runs, random order each time\n")
print(f"{'test':<40}{'failed':>8}  rate")
for name in sorted(seen):
    print(f"{name:<40}{fails[name]:>8}  {fails[name] / seen[name]:5.0%}")

$ python flake_rate.py 100
100 runs, random order each time

test                                      failed  rate
test_add_a_wrap                               55    55%
test_big_orders_get_ten_percent_off           10    10%
test_delivery_estimate                        42    42%
test_kitchen_is_open_now                       0     0%
test_new_basket_is_empty                      59    59%

It took 73 seconds. Four of the five tests fail some of the time, for three different reasons:

• test_new_basket_is_empty (59%) and test_add_a_wrap (55%) share one basket for the whole test run. Whichever test runs before them leaves dishes in it.
• test_delivery_estimate (42%) sleeps for 0.1 seconds and hopes the answer has arrived. The answer takes 50 to 150 milliseconds.
• test_big_orders_get_ten_percent_off (10%) picks a random basket size each run. We'll come back to this one, because it's not a test problem.

test_kitchen_is_open_now passed all 100 times. We ran this at 11:28 in the morning, India time.

Step 3: Run it somewhere else

Most CI machines run on UTC. At the same moment, in UTC, it was 06:00:

$ TZ=UTC python flake_rate.py 20
20 runs, random order each time

test                                      failed  rate
test_add_a_wrap                               14    70%
test_big_orders_get_ten_percent_off            1     5%
test_delivery_estimate                         5    25%
test_kitchen_is_open_now                      20   100%
test_new_basket_is_empty                      13    65%

The kitchen test failed every time. On a developer's laptop in India it passes during the day. On the CI server it fails until 11:00 UTC, which is 16:30 in India. Depending on when people look, it's "always green", "always red" or "flaky". It's the same problem as the Part 14 test: it depends on the real clock.

Step 4: Reproduce a failure

A flaky failure you can't repeat is hard to fix. pytest-randomly prints the seed it used for the order at the start of each run:

$ pytest -p randomly
Using --randomly-seed=2365621843
...
FAILED tests/test_basket.py::test_delivery_estimate - assert None == 25
FAILED tests/test_basket.py::test_new_basket_is_empty - AssertionError: asser...
FAILED tests/test_basket.py::test_add_a_wrap - assert 2430 == 180
========================= 3 failed, 2 passed in 0.17s ==========================

Run it again with that seed, and you get the same order and the same failures:

$ pytest -p randomly --randomly-seed=2365621843 -v
Using --randomly-seed=2365621843
tests/test_basket.py::test_big_orders_get_ten_percent_off PASSED         [ 20%]
tests/test_basket.py::test_delivery_estimate FAILED                      [ 40%]
tests/test_basket.py::test_new_basket_is_empty FAILED                    [ 60%]
tests/test_basket.py::test_kitchen_is_open_now PASSED                    [ 80%]
tests/test_basket.py::test_add_a_wrap FAILED                             [100%]
...
========================= 3 failed, 2 passed in 0.17s ==========================

Now the cause is visible: the big-orders test ran first and left 15 wraps in the shared basket, so "add a wrap" saw ₹2,430 instead of ₹180. pytest-randomly also resets Python's random module with the same seed before each test, so even the delivery-time delay repeated exactly.

Step 5: The tempting shortcut: retries

The usual reaction to flaky tests is to retry failures automatically. We tried it, 20 times, with --reruns 2:

$ pytest -p randomly --reruns 2 -q
2 failed, 3 passed, 4 rerun in 0.41s
1 failed, 4 passed, 2 rerun in 0.23s
1 failed, 4 passed, 3 rerun in 0.36s
...
green runs with reruns: 1 / 20

Retries made 1 run in 20 green. A rerun happens in the same process, so the basket is still full from the earlier test, and the retry fails the same way.

Where retries do turn a suite green, they make things worse in a different way: the failures are still happening, but nobody sees them. The test still fails some of the time, and when it's a real bug, it's now hidden. If you use retries at all, record every retried failure and count them, so flakiness stays visible.

Step 6: Fix each cause

Shared state. Each test gets its own basket. That meant changing the code, not just the tests: a module-level basket that lives for the whole process was the real problem. The basket becomes a class, and a fixture gives every test a new one:

class Basket:
    def __init__(self):
        self._items: list[str] = []

    def add(self, dish: str) -> None:
        self._items.append(dish)

    def items(self) -> list[str]:
        return list(self._items)

    def total(self) -> int:
        """Food total in rupees. Orders of 10 or more dishes get 10% off."""
        food = sum(PRICES[d] for d in self._items)
        if len(self._items) > 10:
            food = food * 90 // 100
        return food

@pytest.fixture
def basket():
    return Basket()  # a new, empty basket for every test

Timing. Don't sleep and hope. Wait for the thing you need, with a time limit. The service sets an event when the answer arrives:

class EtaService:
    def __init__(self):
        self.minutes: int | None = None
        self.ready = threading.Event()

    def start(self) -> None:
        def work():
            time.sleep(random.uniform(0.05, 0.15))  # the maps service takes 50-150 ms
            self.minutes = 25
            self.ready.set()
        threading.Thread(target=work, daemon=True).start()

def test_delivery_estimate():
    eta = EtaService()
    eta.start()
    assert eta.ready.wait(timeout=2), "the maps service didn't answer within 2 seconds"
    assert eta.minutes == 25

The test now continues the moment the answer arrives, after 50 to 150 ms, which on average is no slower than the old sleep. It only fails if the service really takes more than 2 seconds.

The clock. Pass the time in. The function already accepted now. The test just didn't use it:

@pytest.mark.parametrize("hour, is_open", [(10, False), (11, True), (22, True), (23, False)])
def test_kitchen_hours(hour, is_open):
    assert is_kitchen_open(datetime(2026, 11, 12, hour, 30)) is is_open

That's also a better test: it checks both sides of opening and closing time, instead of whatever time it happens to be.

Randomness. Replace the random basket size with the sizes that matter: either side of the discount boundary, and a big one:

@pytest.mark.parametrize("n", [9, 10, 11, 20])
def test_big_order_discount(basket, n):
    for _ in range(n):
        basket.add("Paneer wrap")
    expected = 180 * n * 90 // 100 if n >= 10 else 180 * n
    assert basket.total() == expected

Step 7: The flaky test that was a real bug

With the test fixes in place, the 100-run measurement on UTC:

$ TZ=UTC python flake_rate.py 100
test                                      failed  rate
test_add_a_wrap                                0     0%
test_big_order_discount[10]                  100   100%
test_big_order_discount[11]                    0     0%
test_big_order_discount[20]                    0     0%
test_big_order_discount[9]                     0     0%
test_delivery_estimate                         0     0%
test_kitchen_hours[10-False]                   0     0%
...

FAILED tests/test_basket.py::test_big_order_discount[10] - assert 1800 == 1620

The 10% "flaky" test wasn't flaky at all. The code says if len(self._items) > 10, but the rule, in its own comment, is "10 or more dishes get 10% off". An order of exactly 10 dishes was charged ₹1,800 instead of ₹1,620. The random basket size picked exactly 10 about one time in eleven, so the bug showed up as a test that "sometimes fails".

This is why flaky tests deserve attention instead of re-runs: some of them are real bugs that only happen sometimes. Race conditions, time zones, boundary values and rare data all look like flakiness from the outside.

The fix is one character:

-        if len(self._items) > 10:
+        if len(self._items) >= 10:

$ TZ=UTC python flake_rate.py 100
test                                      failed  rate
test_add_a_wrap                                0     0%
test_big_order_discount[10]                    0     0%
test_big_order_discount[11]                    0     0%
test_big_order_discount[20]                    0     0%
test_big_order_discount[9]                     0     0%
test_delivery_estimate                         0     0%
test_kitchen_hours[10-False]                   0     0%
test_kitchen_hours[11-True]                    0     0%
test_kitchen_hours[22-True]                    0     0%
test_kitchen_hours[23-False]                   0     0%
test_new_basket_is_empty                       0     0%

And 30 more runs on India time: 0 failures. Eleven tests, 130 runs, no failures.

Common causes, and their fixes

A team policy that works

1. Measure. Track each test's pass rate across runs, not just whether the latest build was green.
2. Quarantine, with an owner and a date. Move a known flaky test out of the blocking suite, keep running it, and give it an owner and a deadline. A quarantine without a deadline is a deletion.
3. Find the cause before fixing the test. Run it with the failing seed, at the failing time, in the failing environment. One in four of our flaky tests was a real bug.
4. Don't let retries hide failures. If you retry, report every retried failure.
5. Run in random order in CI, so shared state is found when it's added, not months later.

Common mistakes

• Re-running until green. It hides real bugs, and in our run it didn't even work: 1 green run in 20.
• Raising sleeps. sleep(0.1) becomes sleep(1). The test gets slower and is still flaky on a bad day.
• Testing with the real clock. The test passes where it was written and fails at night or on UTC.
• Random test data without a seed. You can't reproduce the failure, and you may be hiding a boundary bug.
• Deleting the test. The flakiness might be the only sign of a real race or boundary bug.
• Running in one fixed order. Shared state stays hidden until someone adds a test in the wrong place.

What's next

In Part 17 of Testing in Practice we look at property-based testing with Hypothesis: instead of choosing examples, describe what must always be true, and let the tool search for the inputs that break it. In a quick check on the buggy code, it found the 10-dish bug in all five runs, in under a second each.

ProvaLab's test management system keeps the history of every test across runs, so a test that passes, fails and passes again on the same code shows up as a pattern, not as a red build someone re-ran. Results arrive from pytest, Cypress, Playwright and other frameworks through ProvaLab's plugins, and you can re-run just the failed tests of a run from TMS, with the re-run recorded as a run of its own. See it in the two-minute product tour.

• The German for "Change address" doesn't fit its button.
• A price shows as ₹125,000 when an Indian reader expects ₹1,25,000.
• "11/12/2026" means 12 November to an American and 11 December to almost everyone else.
• Arabic is written right to left, and it has six plural forms.

Localization testing checks that the app works and reads correctly in each language and region it supports. It's usually split in two:

• Internationalization (i18n) testing: is the code ready for other languages? No text built into the code, formats taken from the locale, layouts that grow and flip. Developers and testers can check this, often before any translation exists.
• Linguistic testing: are the translations right, natural and appropriate? That needs native speakers.

This post is mostly about the first kind, because it can be automated. We tested a FoodNow order page in Hindi, German and Arabic with Playwright and pytest. Six kinds of bug were planted in it. The tests found them all, and one we didn't plant. Every result below is from a real run.

The translations in this post were written for this example. Before you ship translations, have native speakers review them. That's the linguistic testing a script can't do.

The page and its translations

The page shows a FoodNow order: the delivery address with a button to change it, the number of items, the delivery fee, the total, and the delivery date. The language comes from ?lang= in the address. The translations live in site/strings.js:

const STRINGS = {
  en: {
    title: "Your order", fee: "Delivery fee", total: "Total", place: "Place order", address: "Change address",
    item_one: "{n} item", item_other: "{n} items",
    deliver_on: "Delivery on {date}",
  },
  hi: {
    title: "आपका ऑर्डर", fee: "डिलीवरी शुल्क", total: "कुल", place: "ऑर्डर करें", address: "पता बदलें",
    item_one: "{n} आइटम", item_other: "{n} आइटम",
    deliver_on: "डिलीवरी {date} को",
  },
  de: {
    title: "Ihre Bestellung", fee: "Liefergebühr", total: "Gesamt", place: "Bestellung aufgeben", address: "Lieferadresse ändern",
    item_one: "{n} Artikel", item_other: "{n} Artikel",
    deliver_on: "Lieferung am {date}",
  },
  ar: {
    title: "طلبك", fee: "رسوم التوصيل", total: "المجموع", place: "تأكيد الطلب", address: "تغيير العنوان",
    item_zero: "لا توجد عناصر", item_one: "عنصر واحد", item_two: "عنصران",
    item_few: "{n} عناصر", item_many: "{n} عنصرًا", item_other: "{n} عنصر",
    deliver_on: "التوصيل في {date}",
  },
};

Arabic has six plural forms: zero, one, two, few (3–10), many (11–99) and other (for example 100). English has two. Hindi and German also have two, but Hindi counts 0 as "one" and German doesn't. The Unicode CLDR project publishes these rules for every language, and browsers include them.

The first version of the page's script, with its bugs:

const money = paise => "₹" + (paise / 100).toLocaleString("en-US");
document.getElementById("count").textContent = t(items === 1 ? "item_one" : "item_other", { n: items });
document.getElementById("total").textContent = money(totalPaise);
const when = new Date(2026, 10, 12);  // 12 November 2026
document.getElementById("when").textContent = t("deliver_on", { date: when.toLocaleDateString("en-US") });

And in the HTML:

<div class="row"><span>Delivery fee</span><span id="fee"></span></div>

.addr button { width: 130px; white-space: nowrap; overflow: hidden; }

Six bugs, all common in real apps:

1. "Delivery fee" is typed into the HTML instead of coming from the translations.
2. Money is always formatted the American way.
3. Plurals use English's rule: one, or not one.
4. The date is always American: month first.
5. The page never says it's Arabic, or that Arabic runs right to left.
6. The address button has a fixed width and hides anything that doesn't fit.

Step 1: Set up

mkdir foodnow-l10n && cd foodnow-l10n
python -m venv .venv && source .venv/bin/activate
pip install pytest==9.2.0 playwright==1.64.0
playwright install chromium
mkdir -p site tests shots

To see Hindi and Arabic text in screenshots, the machine needs fonts for them. We used Noto Sans Devanagari and Noto Sans Arabic. Without them, the browser draws empty boxes, and your screenshots can't show what users see.

Step 2: Pseudo-localization

You don't have to wait for translations to start testing. Pseudo-localization turns every English string into a fake language that's still readable, so problems stand out before any translator is involved:

// Pseudo-localization: accented, longer, and bracketed, so untranslated or cut-off text stands out.
const ACCENTS = { a: "å", e: "é", i: "ï", o: "ö", u: "ü", A: "Å", E: "É", I: "Ï", O: "Ö", U: "Ü", c: "ç", n: "ñ" };
function pseudo(text) {
  const accented = text.replace(/\{[^}]+\}|./g, ch => ch.startsWith("{") ? ch : (ACCENTS[ch] || ch));
  const pad = "~".repeat(Math.ceil(text.length * 0.4));
  return `[${accented}${pad}]`;
}
STRINGS["en-XA"] = Object.fromEntries(Object.entries(STRINGS.en).map(([k, v]) => [k, pseudo(v)]));

"Change address" becomes [Chåñgé åddréss~~~~~~]. Three things happen at once:

• Brackets show where each string starts and ends, so you can see text that's cut off or joined wrongly.
• Accented letters show text that can't handle characters beyond plain English.
• About 40% extra length imitates languages like German, which often run 30% longer than English, and more for short strings.

Any interface text without brackets didn't come from the translations, so it will stay in English in every language. en-XA is a code commonly used for this, including on Android, which has pseudo-locales built in.

Step 3: The tests

tests/conftest.py serves the site folder and opens a page in the right browser locale for each language:

import functools
import http.server
import threading
from pathlib import Path

import pytest
from playwright.sync_api import sync_playwright

SITE = Path(__file__).parents[1] / "site"

# The app's language code, and the browser locale a user of that language would have.
LANGS = {"en": "en-IN", "hi": "hi-IN", "de": "de-DE", "ar": "ar", "en-XA": "en-IN"}


@pytest.fixture(scope="session")
def base_url():
    handler = functools.partial(http.server.SimpleHTTPRequestHandler, directory=SITE)
    server = http.server.ThreadingHTTPServer(("127.0.0.1", 0), handler)
    threading.Thread(target=server.serve_forever, daemon=True).start()
    yield f"http://127.0.0.1:{server.server_port}"
    server.shutdown()


@pytest.fixture(scope="session")
def browser():
    with sync_playwright() as p:
        browser = p.chromium.launch()
        yield browser
        browser.close()


@pytest.fixture
def open_page(browser, base_url):
    pages = []

    def _open(lang, items=3, total=12500000, width=412):
        context = browser.new_context(locale=LANGS[lang], viewport={"width": width, "height": 800})
        page = context.new_page()
        page.goto(f"{base_url}/index.html?lang={lang}&items={items}&total={total}")
        pages.append(context)
        return page

    yield _open
    for c in pages:
        c.close()

locale= sets the browser's language and region, as a real user's phone would. The total is ₹1,25,000 (12,500,000 paise), a lunch for 400 people, big enough to show the Indian way of grouping digits.

And tests/test_l10n.py, one test for each kind of problem:

import pytest


def clean(text: str) -> str:
    """Drop invisible direction marks and turn non-breaking spaces into spaces."""
    return text.replace("‏", "").replace("‎", "").replace("\xa0", " ").strip()


# 1. Pseudo-localization: every piece of interface text must come from the translations.
def test_no_hardcoded_text(open_page):
    page = open_page("en-XA")
    user_data = {"where"}  # the customer's own address is not interface text
    hardcoded = page.evaluate("""userData => [...document.querySelectorAll('main *')]
        .filter(e => e.children.length === 0 && /[A-Za-z]/.test(e.textContent) && !userData.includes(e.id))
        .map(e => e.textContent.trim())
        .filter(t => !t.startsWith('['))""", list(user_data))
    assert hardcoded == []


# 2. Text must fit: nothing cut off, in any language, including pseudo-localized text.
@pytest.mark.parametrize("lang", ["en", "hi", "de", "ar", "en-XA"])
def test_buttons_fit_their_text(open_page, lang):
    page = open_page(lang)
    cut = page.evaluate("""() => [...document.querySelectorAll('button')]
        .filter(b => b.scrollWidth > b.clientWidth)
        .map(b => `${b.textContent} (${b.scrollWidth}px of text in ${b.clientWidth}px)`)""")
    if cut:
        page.screenshot(path=f"shots/cut-{lang}.png")
    assert cut == []


# 3. Money in each language's format.
@pytest.mark.parametrize("lang, total, fee", [
    ("en", "₹1,25,000", "₹40"),
    ("hi", "₹1,25,000", "₹40"),
    ("de", "125.000 ₹", "40 ₹"),
    ("ar", "125,000 ₹", "40 ₹"),
])
def test_money_format(open_page, lang, total, fee):
    page = open_page(lang)
    assert clean(page.text_content("#total")) == total
    assert clean(page.text_content("#fee")) == fee


# 4. Dates nobody can misread: 12 November 2026.
@pytest.mark.parametrize("lang, expected", [
    ("en", "Delivery on 12 November 2026"),
    ("hi", "डिलीवरी 12 नवंबर 2026 को"),
    ("de", "Lieferung am 12. November 2026"),
    ("ar", "التوصيل في 12 نوفمبر 2026"),
])
def test_delivery_date(open_page, lang, expected):
    assert clean(open_page(lang).text_content("#when")) == expected


# 5. Plurals: Arabic has six forms.
@pytest.mark.parametrize("n, expected", [
    (1, "عنصر واحد"), (2, "عنصران"), (3, "3 عناصر"), (11, "11 عنصرًا"), (100, "100 عنصر"),
])
def test_arabic_plurals(open_page, n, expected):
    assert clean(open_page("ar", items=n).text_content("#count")) == expected


# 6. Right-to-left: Arabic pages must be laid out right to left.
def test_arabic_is_right_to_left(open_page):
    page = open_page("ar")
    assert page.evaluate("document.documentElement.dir") == "rtl"
    assert page.evaluate("document.documentElement.lang") == "ar"
    # Where the heading's text actually is (its box is full width either way).
    text_left = page.evaluate("""() => {
        const range = document.createRange();
        range.selectNodeContents(document.getElementById('title'));
        return range.getBoundingClientRect().left;
    }""")
    assert text_left > 200, f"the heading should start on the right, but its text starts at x={text_left:.0f}"

A few details that matter:

• Where the expected formats come from. We asked Chromium's built-in Intl formatting what each locale uses, then checked that it was right for our users before writing it into the tests. In German, the currency sign goes after the number, with a dot for thousands. Chromium writes Arabic amounts with Western digits (125,000) by default. Some Arabic-speaking regions prefer Eastern Arabic digits (١٢٥٬٠٠٠). That's a product decision, so the test records the decision you made.
• clean(). Formatted numbers contain invisible characters: a non-breaking space between "125.000" and "₹", and in Arabic a right-to-left mark. They're correct, but they make string comparisons fail in confusing ways. Strip them in the comparison, not in the app.
• The right-to-left check measures the text, not the box. Our first version checked the heading element's position. A heading's box is the full width of the page in either direction, so that check could never fail. Measuring where the text itself is fixed it: on the buggy page the heading's text starts at x=36, on the fixed page at x=327.

Step 4: Run it

$ pytest tests -q --tb=line
E   AssertionError: assert ['Delivery fee'] == []
E   AssertionError: assert ['Change addr...xt in 128px)'] == []
E   AssertionError: assert ['Lieferadres...xt in 128px)'] == []
E   AssertionError: assert ['[Chåñgé ådd...xt in 128px)'] == []
E   AssertionError: assert '₹125,000' == '₹1,25,000'
E   AssertionError: assert '₹125,000' == '₹1,25,000'
E   AssertionError: assert '₹125,000' == '125.000 ₹'
E   AssertionError: assert '₹125,000' == '125,000 ₹'
E   AssertionError: assert 'Delivery on 11/12/2026' == 'Delivery on 12 November 2026'
E   AssertionError: assert 'डिलीवरी 11/12/2026 को' == 'डिलीवरी 12 नवंबर 2026 को'
E   AssertionError: assert 'Lieferung am 11/12/2026' == 'Lieferung am...November 2026'
E   AssertionError: assert 'التوصيل في 11/12/2026' == 'التوصيل في 12 نوفمبر 2026'
E   AssertionError: assert '2 عنصر' == 'عنصران'
E   AssertionError: assert '3 عنصر' == '3 عناصر'
E   AssertionError: assert '11 عنصر' == '11 عنصرًا'
E   AssertionError: assert '' == 'rtl'
16 failed, 4 passed in 4.66s

Sixteen failures on one small page. What they mean for users:

• "Delivery fee" stays in English in every language. Pseudo-localization found it with no translator involved.
• The address button cuts its text off. In German, "Lieferadresse ändern" needs 162 pixels and gets 128. Pseudo-localized, it needs 191. And the one we didn't plant: the English button is cut off too, by 2 pixels. "Change address" needs 130 pixels in the font on our machine. It probably fitted in whatever font the button was designed with. A fixed width with hidden overflow fails as soon as the font, the language or the user's text size changes.
• ₹125,000 is wrong for Indian readers, who group digits as ₹1,25,000 (one lakh, twenty-five thousand). In German, it reads as if the comma were a decimal point: one hundred twenty-five rupees.
• 11/12/2026 means 11 December to an Indian, German or Arabic reader. The delivery is on 12 November. A delivery date that's wrong by a month is a real bug, not a cosmetic one.
• Arabic plurals were only right for 1 and 100, by luck: English's "one or other" rule happens to match those two of Arabic's six forms.
• No right-to-left layout. The Arabic text is there, but the page runs left to right: the heading on the left, numbers and labels on the wrong sides.

The German page showed four of these at once:

Step 5: Fix it with the platform, not by hand

Browsers already know every one of these rules, through the Intl APIs. The fix is mostly to stop overriding them:

-  .addr button { width: 130px; white-space: nowrap; overflow: hidden; ... }
+  .addr button { min-width: 130px; max-width: 60%; ... }
...
-    <div class="row"><span>Delivery fee</span><span id="fee"></span></div>
+    <div class="row"><span id="fee-label"></span><span id="fee"></span></div>
...
+  const LOCALES = { en: "en-IN", hi: "hi-IN", de: "de-DE", ar: "ar", "en-XA": "en-IN" };
+  const locale = LOCALES[lang] || "en-IN";
+  document.documentElement.lang = lang;
+  document.documentElement.dir = lang === "ar" ? "rtl" : "ltr";
+
-  const money = paise => "₹" + (paise / 100).toLocaleString("en-US");
+  const moneyFormat = new Intl.NumberFormat(locale, { style: "currency", currency: "INR", maximumFractionDigits: 0 });
+  const money = paise => moneyFormat.format(paise / 100);
+  const plural = new Intl.PluralRules(locale);
...
-  document.getElementById("count").textContent = t(items === 1 ? "item_one" : "item_other", { n: items });
+  const form = plural.select(items);
+  document.getElementById("count").textContent = t(s[`item_${form}`] ? `item_${form}` : "item_other", { n: items });
+  document.getElementById("fee-label").textContent = t("fee");
...
-  ... t("deliver_on", { date: when.toLocaleDateString("en-US") });
+  ... t("deliver_on", { date: when.toLocaleDateString(locale, { dateStyle: "long" }) });

• Buttons grow and wrap instead of hiding text. Design for the longest language, not for English.
• Every label comes from the translations, including "Delivery fee".
• Intl.NumberFormat formats money for the locale: ₹1,25,000 in India, 125.000 ₹ in Germany.
• Intl.PluralRules picks the right plural form, so Arabic gets all six.
• Dates use the long format, "12 November 2026", which nobody can misread. Avoid all-number dates in anything a customer reads.
• lang and dir on the page. dir="rtl" flips the whole layout for Arabic. lang lets screen readers pronounce the text correctly, and lets the browser choose the right fonts.

$ pytest tests -q
....................                                                     [100%]
20 passed in 4.34s

What automated tests can't check

The tests above check that the app is ready for other languages. They can't tell you whether:

• The translation is right. "Bestellung aufgeben" is the usual German for "place order". A machine translation might produce something technically correct that no German shop would write.
• The tone fits. Formal or informal "you" (German Sie or du, Hindi आप or तुम) is a brand decision.
• Images, colours and examples make sense in the region. A dish, a gesture or a holiday that's normal in one country can confuse or offend in another.
• Legal and regional rules are met: address formats, tax display, required notices.

That's linguistic and cultural testing, done by native speakers reading the real app, ideally on the screens and in the situations real users have. Give them the app, not a spreadsheet of strings: a translation can be right in a spreadsheet and wrong on the button.

Common mistakes

• Waiting for translations to start testing. Pseudo-localization finds hardcoded text and cut-off layouts on day one.
• Building text from pieces. "You have " + n + " items" can't be translated: word order and plurals differ. Translate whole sentences with placeholders.
• English plural logic. n === 1 ? one : many is wrong for Arabic, Russian, Polish and many more. Use Intl.PluralRules.
• Formatting numbers and dates by hand. Use Intl.NumberFormat and Intl.DateTimeFormat with the user's locale.
• Fixed widths with hidden overflow. They cut off German, pseudo-localized text and, in our run, English too.
• Testing right-to-left by checking an element's box. Boxes are often full width. Check where the text actually is, and look at the screen.
• No fonts on the test machine. Your screenshots show empty boxes, and you can't see what users see.

What's next

In Part 16 of Testing in Practice we look at flaky tests: why a test passes and fails without a code change, how to find which tests are flaky, and how to fix the causes, with the clock test from Part 14 and the race from Part 12 as real examples.

ProvaLab's QA Infrastructure runs browser tests in the cloud, with a live view while they run, and ProvaLab's pytest plugin sends each localized test's result, one per language, to the ProvaLab test management system. Each language's history stays separate, so a regression in Arabic shows up as Arabic, not as one failure among hundreds. See it in the two-minute product tour.

Property-based testing turns that around. You describe what must be true for every input, a property, and a tool generates hundreds of inputs to try to break it. When it finds one, it shrinks it to the smallest input that still fails, so you get "1 paisa between 2 people" instead of "₹8,471,293.17 between 37 people".

The best-known tool for Python is Hypothesis. This post uses it on FoodNow's money code: discounts, splitting a group order's bill, and showing prices in Indian format. The code has three bugs, and seven example tests that all pass. Every result below is from a real run, with Hypothesis 6.170.15.

The code

foodnow/money.py. All amounts are in paise (₹1 = 100 paise):

"""FoodNow money helpers. All amounts are in paise (₹1 = 100 paise)."""


def discounted_food_total(dishes: int, price: int) -> int:
    """Orders of 10 or more dishes get 10% off."""
    food = dishes * price
    if dishes > 10:
        food = food * 90 // 100
    return food


def split_bill(total: int, people: int) -> list[int]:
    """Split a group order's total between colleagues."""
    share = round(total / people)
    return [share] * people


def format_inr(paise: int) -> str:
    """₹1,25,000.50 - Indian digit grouping: the last three digits, then pairs."""
    rupees, p = divmod(paise, 100)
    digits = str(rupees)
    head, tail = digits[:-3], digits[-3:]
    groups = []
    while len(head) > 2:
        groups.insert(0, head[-2:])
        head = head[:-2]
    if head:
        groups.insert(0, head)
    whole = ",".join(groups + [tail])
    return f"₹{whole}.{p}" if p else f"₹{whole}"


def parse_inr(text: str) -> int:
    whole, _, frac = text.removeprefix("₹").replace(",", "").partition(".")
    return int(whole) * 100 + (int(frac) if frac else 0)

split_bill is for group ordering, the feature that Part 14 found had no tests. format_inr uses the Indian digit grouping from Part 15.

Step 1: The example tests

These are the tests the team already had, in tests/test_examples.py:

"""The example-based tests the team already had. All of them pass."""
from foodnow.money import discounted_food_total, format_inr, parse_inr, split_bill


def test_small_order_no_discount():
    assert discounted_food_total(2, 18000) == 36000


def test_big_order_discount():
    assert discounted_food_total(12, 18000) == 194400


def test_split_evenly():
    assert split_bill(90000, 3) == [30000, 30000, 30000]


def test_split_between_two():
    assert split_bill(41000, 2) == [20500, 20500]


def test_format_lakh():
    assert format_inr(12500000) == "₹1,25,000"


def test_format_with_paise():
    assert format_inr(12500050) == "₹1,25,000.50"


def test_parse():
    assert parse_inr("₹1,25,000.50") == 12500050

pip install pytest==9.2.0 hypothesis==6.170.15

$ pytest tests/test_examples.py -q
.......                                                                  [100%]
7 passed in 0.65s

Seven reasonable examples, all green, three bugs.

Step 2: Write properties

A property says what's always true. For money code, good properties come from the rules and from common sense:

• 10 or more dishes get 10% off; fewer don't.
• When you split a bill, the shares add up to the total. No paisa appears or disappears.
• The split is fair: nobody pays more than one paisa more than anyone else.
• If you format an amount and parse the text back, you get the same amount.

tests/test_properties.py:

"""Properties: things that must be true for every input, not just the examples we thought of."""
from hypothesis import given, strategies as st

from foodnow.money import discounted_food_total, format_inr, parse_inr, split_bill

paise = st.integers(min_value=0, max_value=10**10)          # up to ₹10 crore
people = st.integers(min_value=1, max_value=50)


@given(dishes=st.integers(min_value=1, max_value=100), price=st.integers(min_value=100, max_value=100_000))
def test_ten_or_more_dishes_get_ten_percent_off(dishes, price):
    expected = dishes * price * 90 // 100 if dishes >= 10 else dishes * price
    assert discounted_food_total(dishes, price) == expected


@given(total=paise, n=people)
def test_split_adds_up_to_the_total(total, n):
    assert sum(split_bill(total, n)) == total


@given(total=paise, n=people)
def test_split_is_fair(total, n):
    shares = split_bill(total, n)
    assert len(shares) == n
    assert max(shares) - min(shares) <= 1  # nobody pays more than one paisa extra


@given(amount=paise)
def test_format_then_parse_gives_the_same_amount(amount):
    assert parse_inr(format_inr(amount)) == amount

@given tells Hypothesis to call the test many times (100 by default), each time with values drawn from the strategies: here, whole numbers in a range. Hypothesis doesn't pick them uniformly. It favours values that tend to break code: zero, the ends of the range, small numbers, and numbers it has seen cause trouble.

Step 3: Run it, and read it carefully

$ pytest tests/test_properties.py -q
E       assert 0 == 1
E        +  where 0 = sum([0, 0])
E        +    where [0, 0] = split_bill(1, 2)
E       Failing test case: test_split_adds_up_to_the_total(
E           total=1,
E           n=2,
E       )
1 failed, 3 passed in 0.36s

One real bug, shrunk beautifully: 1 paisa split between 2 people gives [0, 0]. A paisa disappears. round(0.5) is 0 in Python. With bigger numbers it's the same problem: ₹1,000 split between 3 people becomes three shares of ₹333.33, and a paisa is lost; other amounts round up and charge the group more than the bill. The example tests used amounts that divide exactly.

But look at what passed. Three bugs, and only one property failed. That needs explaining before you trust the green ones.

The fairness property passed because [share] * people is perfectly fair. Every share is the same. It's just wrong. A property can only check what it says, so you need more than one property per function: here, "adds up" and "is fair" together.

The round-trip property passed with a real bug in it. One paisa is formatted as ₹0.1, which anyone would read as 10 paise. But parse_inr("₹0.1") reads the 1 after the dot as 1 paisa, so the round trip gives back the right number. Formatting and parsing share the same mistake, so the round trip can't see it. Round-trip properties are excellent at finding asymmetry between two functions, and blind when both are wrong in the same way.

The discount property passed even though 10 dishes don't get the discount. Hypothesis tries the ends of the range (1 and 100) and many values in between, but 10 is just one value in the middle, and the second input, the price, spreads its effort further. We measured it: 50 runs, each from scratch, with these strategies. Hypothesis found the bug in 20 of the 50. With a narrower range and a fixed price, 1 to 50 dishes, it found it in 50 of 50. Boundaries that live inside your rules, rather than at the edges of a range, are worth stating yourself.

Step 4: Make the properties honest

Two changes. An explicit example pins the boundary, so it's checked on every run. And a property that compares the formatter with an independent, obviously correct description of the format, instead of with its own parser:

import re

from hypothesis import example, given, strategies as st


@given(dishes=st.integers(min_value=1, max_value=100), price=st.integers(min_value=100, max_value=100_000))
@example(dishes=9, price=18000)
@example(dishes=10, price=18000)  # the boundary in the rule: always check it
def test_ten_or_more_dishes_get_ten_percent_off(dishes, price):
    expected = dishes * price * 90 // 100 if dishes >= 10 else dishes * price
    assert discounted_food_total(dishes, price) == expected


@given(amount=paise)
def test_format_shows_two_digit_paise_and_indian_grouping(amount):
    text = format_inr(amount)
    rupees, p = divmod(amount, 100)
    assert text.replace(",", "") == f"₹{rupees}" + (f".{p:02d}" if p else "")
    assert re.fullmatch(r"₹(\d{1,2},)?(\d{2},)*\d{1,3}(\.\d{2})?", text), text

The new formatting property says two things: without the commas, the text is the rupees and two-digit paise. And the commas follow the Indian pattern: a group of three at the end, then pairs.

One more change, in tests/conftest.py:

from hypothesis import settings

# Don't replay saved failures from earlier runs, so every run shows what Hypothesis finds from scratch.
settings.register_profile("fresh", database=None)
settings.load_profile("fresh")

Normally Hypothesis saves every failing input in a .hypothesis folder and tries it first on the next run. That's a feature: a bug it found once is checked every time after that, until it's fixed. But it also means "it found the bug again" may only mean "it remembered". For this post we turned it off, so each run starts from nothing. In your own project, keep the database. Commit it, or share it in CI, so everyone benefits from what Hypothesis has found.

Three runs, each from scratch:

3 failed, 2 passed in 0.24s
3 failed, 2 passed in 0.26s
3 failed, 2 passed in 0.22s

E       assert 180000 == 162000
E        +  where 180000 = discounted_food_total(10, 18000)
E       Failing explicit example: test_ten_or_more_dishes_get_ten_percent_off(
E           dishes=10,
E           price=18000,
E       )

E       assert 0 == 1
E        +  where 0 = sum([0, 0])
E        +    where [0, 0] = split_bill(1, 2)
E       Failing test case: test_split_adds_up_to_the_total(
E           total=1,
E           n=2,
E       )

E       AssertionError: assert '₹0.1' == '₹0.01'
E       Failing test case: test_format_shows_two_digit_paise_and_indian_grouping(
E           amount=1,
E       )

All three bugs, every time, each shrunk to the smallest case: 10 dishes at ₹180 charged ₹1,800 instead of ₹1,620; 1 paisa split into nothing; 1 paisa shown as ₹0.1.

Step 5: Fix the code

-    if dishes > 10:
+    if dishes >= 10:
...
-    share = round(total / people)
-    return [share] * people
+    base, extra = divmod(total, people)
+    return [base + 1] * extra + [base] * (people - extra)  # the first `extra` people pay one paisa more
...
-    return f"₹{whole}.{p}" if p else f"₹{whole}"
+    return f"₹{whole}.{p:02d}" if p else f"₹{whole}"

The split now divides exactly: ₹1,000 between 3 people is ₹333.34, ₹333.33 and ₹333.33, which adds up to ₹1,000 and is fair to within a paisa.

$ pytest tests -q
............                                                             [100%]
12 passed in 0.26s

Run three times from scratch: 12 passed each time. That's 7 examples, plus 5 properties that each tried 100 inputs, in about a quarter of a second.

When to use property-based testing

It's a poorer fit for UI flows and for code whose correct output you can only describe by computing it the same way the code does. If the property repeats the code's logic, it shares the code's bugs.

Common mistakes

• One property per function. Our fairness property passed a split that lost money. Combine properties that check different things.
• Trusting round trips alone. Format and parse were wrong together. Add a check against an independent description.
• Assuming Hypothesis tries every boundary. It tries the ends of ranges. Boundaries inside your rules need @example. Ours was found in only 20 runs out of 50 without it.
• Mistaking replay for discovery. The .hypothesis database replays old failures. Great in daily use, but don't count it as proof that a new test finds the bug.
• Repeating the code in the property. expected = round(total / n) * n would agree with the bug. Describe what must be true, not how to compute it.
• Generating huge ranges by habit. Choose strategies close to real inputs, and add edge cases as examples.

What's next

In Part 18 of Testing in Practice we run the same FoodNow tests on three browser engines, Chromium, Firefox and WebKit (the engine behind Safari), and look at what differs between them.

ProvaLab's pytest plugin reports Hypothesis tests like any other pytest test, so a property that fails shows up in the ProvaLab test management system with its failure message, and in the test's history across runs. See it in the two-minute product tour.

This guide starts with a small suite of the kind that builds up after a busy sprint. It has six tests, 81% line coverage, and catches none of the three bugs in the code. Then we rebuild it, one pytest feature at a time, and look at what each one catches. Every command was run as written, with these versions:

The code: FoodNow's bill (5 minutes)

FoodNow is the food delivery app we use as the example across our posts. This is its bill maths. Money is kept in paise (₹1 = 100 paise) as integers, so there are no float rounding surprises. The business rules:

• GST is 5% of the food.
• Delivery is ₹40, and free from ₹499.
• A coupon like SAVE50 takes ₹50 off, but only on orders of ₹199 or more. Below that it is ignored and a warning is logged.

The code has two bugs we planted on purpose. Try to spot them before the tests do.

# foodnow/bill.py
"""FoodNow bill maths. All money is in paise (Rs 1 = 100 paise)."""
import json
import logging
import os
from dataclasses import dataclass, field
from pathlib import Path

log = logging.getLogger("foodnow.bill")

GST_RATE = float(os.environ.get("FOODNOW_GST", "0.05"))
DELIVERY_FEE = 4_000          # Rs 40
FREE_DELIVERY_FROM = 49_900   # Rs 499 and above: free delivery
COUPON_MIN_ORDER = 19_900     # coupons need Rs 199 of food


@dataclass
class Line:
    name: str
    price: int   # paise, for one
    qty: int


@dataclass
class Basket:
    lines: list[Line] = field(default_factory=list)
    coupon_off: int = 0

    def add(self, name: str, price: int, qty: int = 1) -> None:
        if price <= 0:
            raise ValueError("price must be above zero")
        self.lines.append(Line(name, price, qty))

    def apply_coupon(self, code: str) -> None:
        if not code.startswith("SAVE"):
            raise ValueError(f"unknown coupon {code!r}")
        self.coupon_off = int(code[4:]) * 100

    def subtotal(self) -> int:
        return sum(line.price * line.qty for line in self.lines)

    def gst(self) -> int:
        return round(self.subtotal() * GST_RATE)

    def delivery(self) -> int:
        return 0 if self.subtotal() > FREE_DELIVERY_FROM else DELIVERY_FEE

    def discount(self) -> int:
        if self.coupon_off and self.subtotal() < COUPON_MIN_ORDER:
            log.warning("coupon ignored: subtotal %d below %d",
                        self.subtotal(), COUPON_MIN_ORDER)
            return 0
        return min(self.coupon_off, self.subtotal())

    def total(self) -> int:
        return self.subtotal() + self.gst() + self.delivery() - self.discount()

    def total_rupees(self) -> float:
        return self.total() / 100


def write_receipt(basket: Basket, folder: Path, order_id: str) -> Path:
    path = folder / f"receipt-{order_id}.json"
    path.write_text(json.dumps({
        "order_id": order_id,
        "lines": [[l.name, l.price, l.qty] for l in basket.lines],
        "total": basket.total(),
    }))
    return path

Set up a project around it:

mkdir pyguide && cd pyguide
mkdir foodnow tests && touch foodnow/__init__.py
# save the code above as foodnow/bill.py
python -m venv .venv && source .venv/bin/activate
pip install pytest==9.2.0 pytest-cov==7.1.0 pytest-xdist==3.8.0

Version 1: the suite after a busy sprint

Here is a suite like many real ones. Nothing in it is unusual.

# tests/test_bill_v1.py
import pytest

from foodnow.bill import Basket


def test_small_order_pays_delivery():
    basket = Basket()
    basket.add("Veg thali", 17_900)
    assert basket.delivery() == 4_000


def test_big_order_free_delivery():
    basket = Basket()
    basket.add("Veg thali", 17_900, qty=3)
    assert basket.delivery() == 0


def test_total_with_gst_and_delivery():
    basket = Basket()
    basket.add("Veg thali", 17_900)
    basket.add("Butter naan", 6_500, qty=2)
    assert basket.total_rupees() == pytest.approx(364.45)


def test_coupon_takes_money_off():
    basket = Basket()
    basket.add("Veg thali", 17_900, qty=2)
    basket.apply_coupon("SAVE50")
    assert basket.discount() == 5_000


def test_negative_quantity_is_refused():
    basket = Basket()
    with pytest.raises(ValueError):
        basket.apply_coupon("SAVE")
        basket.add("Veg thali", 17_900, qty=-1)


def test_gst_rate_comes_from_settings(monkeypatch):
    monkeypatch.setenv("FOODNOW_GST", "0.12")
    basket = Basket()
    basket.add("Veg thali", 10_000)
    assert basket.gst() == 1_200

Run it with plain pytest:

$ pytest -q tests/test_bill_v1.py
ImportError while importing test module '.../tests/test_bill_v1.py'.
E   ModuleNotFoundError: No module named 'foodnow'
1 error in 0.09s

That error is the first lesson. python -m pytest adds the current folder to the import path; plain pytest does not. Rather than remembering which command to type, tell pytest where your code is in a pytest.ini at the project root:

# pytest.ini
[pytest]
testpaths = tests
pythonpath = .

Now it runs:

$ pytest -q tests/test_bill_v1.py
=================================== FAILURES ===================================
______________________ test_gst_rate_comes_from_settings _______________________
    def test_gst_rate_comes_from_settings(monkeypatch):
        monkeypatch.setenv("FOODNOW_GST", "0.12")
        basket = Basket()
        basket.add("Veg thali", 10_000)
>       assert basket.gst() == 1_200
E       AssertionError: assert 500 == 1200
1 failed, 5 passed in 0.03s

Five pass, one fails, and with coverage turned on (more on that below) the suite reaches 81% of the lines. The team's verdict would be "one flaky settings test, otherwise fine". It is not fine. Three things are wrong with this suite.

The raises test passes for the wrong reason. A line copied from the coupon test, basket.apply_coupon("SAVE"), raises ValueError first (the code does int("")), so basket.add(..., qty=-1) never runs. pytest sees a ValueError and is happy. The basket in fact accepts a quantity of −1.

The settings test can never pass. GST_RATE is read once, when foodnow.bill is imported. By the time the test sets the environment variable, the module has already read it. The product works as written; the test is wrong.

The boundary is never tested. The two delivery tests use ₹179 and ₹537. The rule changes at ₹499, and that is where the bug is.

Step 1: Parametrize the boundary, with names (5 minutes)

Boundary tests are where @pytest.mark.parametrize pays for itself: one test body, one row per value either side of the line. Give each row an id, so a failure tells you which row in words:

# tests/test_bill_v2.py
import json
import logging

import pytest

import foodnow.bill as bill

THALI = 17_900   # Rs 179
NAAN = 6_500     # Rs 65


@pytest.mark.parametrize(
    "subtotal, fee",
    [
        pytest.param(49_800, 4_000, id="Rs 498 pays"),
        pytest.param(49_900, 0, id="Rs 499 free"),
        pytest.param(50_000, 0, id="Rs 500 free"),
    ],
)
def test_delivery_fee_boundary(make_basket, subtotal, fee):
    basket = make_basket(("Party platter", subtotal, 1))
    assert basket.delivery() == fee

make_basket is a fixture, which we write in step 3. The failure reads like a bug report:

FAILED tests/test_bill_v2.py::test_delivery_fee_boundary[Rs 499 free] - Asser...
>       assert basket.delivery() == fee
E       AssertionError: assert 4000 == 0
E        +  where 4000 = delivery()

A ₹499 order is charged ₹40 for delivery. The code says > where the rule says "from ₹499", which is >=. That is planted bug number one.

Step 2: pytest.raises with match, and nothing else in the block (5 minutes)

Two rules make pytest.raises trustworthy. Put only the one call you expect to fail inside the with block. And always pass match=, a regular expression the error message must contain, so the test fails if the right type of error is raised for the wrong reason.

def test_negative_quantity_is_refused(make_basket):
    basket = make_basket()
    with pytest.raises(ValueError, match="quantity"):
        basket.add("Veg thali", THALI, qty=-1)


@pytest.mark.parametrize("code", ["SAVE", "SAVEFIFTY", "FREE50"])
def test_bad_coupon_is_refused(make_basket, code):
    basket = make_basket(("Veg thali", THALI, 2))
    with pytest.raises(ValueError, match="coupon"):
        basket.apply_coupon(code)

>       with pytest.raises(ValueError, match="quantity"):
E       Failed: DID NOT RAISE ValueError

Planted bug number two: a negative quantity goes straight into the basket, and the total goes down.

The coupon test found something we did not plant:

FAILED tests/test_bill_v2.py::test_bad_coupon_is_refused[SAVE]
E       AssertionError: Regex pattern did not match.
E         Expected regex: 'coupon'
E         Actual message: "invalid literal for int() with base 10: ''"
FAILED tests/test_bill_v2.py::test_bad_coupon_is_refused[SAVEFIFTY]
E         Expected regex: 'coupon'
E         Actual message: "invalid literal for int() with base 10: 'FIFTY'"

FREE50 gets our "unknown coupon" message. SAVE and SAVEFIFTY start with SAVE, get past the check, and crash in int(). If the app shows that message to a customer, they see Python internals. A bare pytest.raises(ValueError) passes all three and never tells you. This is the same blind spot that hid the quantity bug in version 1.

Step 3: Fixtures: a factory, a teardown, and conftest.py (5 minutes)

Version 1 starts every test with basket = Basket() and a few add calls. A factory fixture returns a function, so each test still says what is in its basket, in one line:

# tests/conftest.py
import pytest

from foodnow.bill import Basket


@pytest.fixture
def make_basket():
    """Factory: make_basket(("Veg thali", 17_900, 2), ...) -> Basket."""
    def _make(*lines):
        basket = Basket()
        for name, price, qty in lines:
            basket.add(name, price, qty)
        return basket
    return _make


@pytest.fixture
def receipts_dir(tmp_path):
    folder = tmp_path / "receipts"
    folder.mkdir()
    yield folder
    print(f"\nteardown: {len(list(folder.iterdir()))} receipt(s) left in {folder.name}/")

Fixtures in conftest.py are available to every test in that folder and below, without an import. Don't import from conftest.py yourself; if tests share constants, put them in a normal module or keep them in the test file, as we did with THALI and NAAN.

receipts_dir shows the other half of fixtures: everything after yield is teardown, and it runs even when the test fails. It builds on tmp_path, a fixture pytest provides: a fresh, empty folder for each test, so no test ever writes into your project or sees another test's files.

def test_receipt_is_written(make_basket, receipts_dir):
    basket = make_basket(("Veg thali", THALI, 1))
    path = bill.write_receipt(basket, receipts_dir, "A1001")
    assert json.loads(path.read_text())["total"] == basket.total()

When you're not sure which fixture runs when, --setup-show prints the order. S is session scope, F is function scope:

$ pytest -q --setup-show -s -k receipt
SETUP    S tmp_path_factory
        SETUP    F make_basket
        SETUP    F tmp_path (fixtures used: tmp_path_factory)
        SETUP    F receipts_dir (fixtures used: tmp_path)
        tests/test_bill_v2.py::test_receipt_is_written (fixtures used: make_basket, receipts_dir, request, tmp_path, tmp_path_factory) .
teardown: 1 receipt(s) left in receipts/

        TEARDOWN F receipts_dir
        TEARDOWN F tmp_path
        TEARDOWN F make_basket
TEARDOWN S tmp_path_factory

Setup runs in dependency order, and teardown in reverse.

Step 4: monkeypatch and caplog: patch the right thing, check the log (5 minutes)

The version 1 settings test set an environment variable the code had already read. monkeypatch can set an attribute instead, and puts the original back after the test:

def test_gst_rate_comes_from_settings(make_basket, monkeypatch):
    monkeypatch.setattr(bill, "GST_RATE", 0.12)
    basket = make_basket(("Veg thali", 10_000, 1))
    assert basket.gst() == 1_200

The general rule: patch the name where the code looks it up, at the time it looks it up. Here that is the module attribute bill.GST_RATE, not the environment. (If the rate must change without a restart, that is a product change: read the variable inside gst(). Then monkeypatch.setenv works.)

caplog captures log records, so you can test the "ignored, with a warning" rule instead of only the "ignored" half:

def test_small_order_coupon_is_ignored_with_a_warning(make_basket, caplog):
    basket = make_basket(("Butter naan", NAAN, 1))
    basket.apply_coupon("SAVE50")
    with caplog.at_level(logging.WARNING, logger="foodnow.bill"):
        assert basket.discount() == 0
    assert "coupon ignored" in caplog.text

A word on pytest.approx and money

Version 1 compared rupees as floats with pytest.approx. It worked there, but look at what approx allows by default, a relative tolerance of one in a million:

>>> 0.1 + 0.2 == 0.3
False
>>> 100_000.09 == pytest.approx(100_000.00)
True
>>> 350.01 == pytest.approx(350.00)
False

On a ₹1,00,000 catering order, approx accepts a total that is 9 paise off. Use approx for real measurements (an average rating, a percentage, a time). For money, keep integers in paise and compare with ==, as test_total_in_paise does in the full file.

Step 5: Markers, and why --strict-markers belongs in every project (3 minutes)

Markers tag tests so you can choose what to run. We register slow in pytest.ini and mark the one heavy test. It checks that no coupon value makes a total negative, over about 118,000 baskets:

@pytest.mark.slwo
def test_total_never_negative_for_any_coupon(make_basket):
    for rupees in range(1, 2_000):
        for qty in range(1, 60):
            basket = make_basket(("Butter naan", NAAN, qty))
            basket.coupon_off = rupees * 100
            assert basket.total() >= 0

Spot the typo: slwo. The fast lane, -m "not slow", still runs the test, because nothing is marked slow:

$ pytest -q -m "not slow" --durations=2
PytestUnknownMarkWarning: Unknown pytest.mark.slwo - is this a typo?
0.57s call     tests/test_bill_v2.py::test_total_never_negative_for_any_coupon
4 failed, 9 passed, 1 warning in 0.62s

Nothing deselected, and a warning most people scroll past. On a real suite, that is how the "quick" job slowly stops being quick. Make unknown markers an error:

# pytest.ini
[pytest]
testpaths = tests
pythonpath = .
addopts = --strict-markers
markers =
    slow: takes more than a second; skip with -m "not slow"

$ pytest -q -m "not slow"
ERROR tests/test_bill_v2.py - Failed: 'slwo' not found in `markers` configuration option
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!

Step 6: Fix the code, and rerun only what failed (3 minutes)

Three small changes to bill.py:

     def add(self, name: str, price: int, qty: int = 1) -> None:
         if price <= 0:
             raise ValueError("price must be above zero")
+        if qty < 1:
+            raise ValueError("quantity must be 1 or more")

     def apply_coupon(self, code: str) -> None:
-        if not code.startswith("SAVE"):
+        rupees = code.removeprefix("SAVE")
+        if rupees == code or not rupees.isdigit():
             raise ValueError(f"unknown coupon {code!r}")
-        self.coupon_off = int(code[4:]) * 100
+        self.coupon_off = int(rupees) * 100

     def delivery(self) -> int:
-        return 0 if self.subtotal() > FREE_DELIVERY_FROM else DELIVERY_FEE
+        return 0 if self.subtotal() >= FREE_DELIVERY_FROM else DELIVERY_FEE

With the typo fixed to @pytest.mark.slow, --lf (last failed) runs only the tests that failed last time. On a big suite, that is the difference between seconds and minutes while you fix things:

$ pytest -q --lf          # before the fix
4 failed, 9 deselected in 0.03s
$ pytest -q --lf          # after the fix
4 passed, 9 deselected in 0.01s
$ pytest -q --lf          # nothing failed last time, so it runs everything
13 passed in 0.59s
$ pytest -q -m "not slow"
12 passed, 1 deselected in 0.02s

Step 7: Coverage shows what you didn't run, not what is right (3 minutes)

pytest-cov measures which lines and branches ran. --cov-branch also counts each side of an if:

$ pytest -q --cov=foodnow --cov-branch --cov-report=term-missing tests/test_bill_v1.py   # buggy code
Name                  Stmts   Miss Branch BrPart  Cover   Missing
foodnow/bill.py          46      7      6      3    81%   30, 35, 49-51, 62-68

$ pytest -q --cov=foodnow --cov-branch --cov-report=term-missing tests/test_bill_v2.py   # fixed code
foodnow/bill.py          49      2      8      1    95%   30, 61

Look at what the version 1 report does not list: the delivery() line. It ran, so it counts as covered, and it held a bug the whole time. Coverage only tells you a line ran, not that any test checked its answer.

The version 2 report is still useful: line 30 is the "price must be above zero" check, which no test reaches. One more parametrized test closes it:

@pytest.mark.parametrize("price", [0, -100])
def test_zero_or_negative_price_is_refused(make_basket, price):
    with pytest.raises(ValueError, match="price"):
        make_basket(("Veg thali", price, 1))

foodnow/bill.py          49      1      8      0    98%   61
15 passed in 2.85s

The one line left is total_rupees(), which only version 1 used. Two side notes from the same runs: coverage slowed the slow test from about 0.55 s to about 2.7 s, because tracing every line costs time; and the report reads 95%, not 100%, and that is fine. Chase the lines that matter, not the number.

Step 8: Parallel runs: measure before you switch them on (2 minutes)

pytest-xdist runs tests across processes with -n. It helps big, slow suites. On this one, it doesn't:

serial  0.82 s   0.82 s   0.86 s
-n 4    1.39 s   1.45 s   1.45 s

Starting four worker processes costs more than the whole 15-test run. The Playwright guide shows the other side: browser tests that went from 27.1 s to 11.6 s with -n 4. Measure on your suite. And before you turn on -n, make sure every test passes alone and in any order; the flaky tests post shows what happens when they don't.

The commands worth remembering

Habits worth keeping

1. Test both sides of every boundary, with parametrize and an id per row that reads like the rule.
2. pytest.raises gets one call and a match=. Without match, any error of that type passes.
3. Patch where the code looks, when it looks. If the value is read at import, set the attribute.
4. Keep money in integers. approx is for measurements.
5. Turn on --strict-markers on day one, in pytest.ini, so a typo fails instead of warning.
6. Treat coverage as a list of untested lines, not as a score. The buggy line in this guide was covered.

What's next

This suite gives one answer on one laptop. On a team, you want every run's results kept, each test linked to the requirement it checks, and the history of a test across runs, so "it failed once last week" is something you can look up. ProvaLab's pytest plugin sends each test's result to a test run in the ProvaLab test management system, and a marker such as @pytest.mark.tms_test_case(id=123) links a test to its test case, so requirement coverage comes from real runs rather than a spreadsheet. See it in the two-minute product tour.

New to pytest? Start with Your first automated test in 15 minutes. For the same ideas applied to browser tests, read Playwright with Python in practice.

Behaviour-driven development (BDD) is a way of finding those questions before the code is written. The product owner, a developer and a tester talk through concrete examples of how a feature should behave. They write the examples down in plain language, as Given / When / Then scenarios. Then the scenarios are automated, so they run as tests every time the code changes.

This post explains BDD and the Gherkin language. Then it applies them to FoodNow's checkout pricing with pytest-bdd. The code has two planted bugs. The scenarios also found a rule that nobody had agreed on, and we hit two traps in the tooling. Every command and line of output below was run while writing this post.

What BDD is, and what it isn't

BDD grew out of test-driven development in the mid-2000s. Dan North coined the name, and the Given/When/Then format came with it. The core idea is that you describe behaviour with examples, not abstract rules, because examples expose the gaps in a rule.

The usual way to start is a short meeting, often called the three amigos: someone who knows what the business wants (the product owner), someone who'll build it (a developer), and someone who thinks about how it could go wrong (a tester). They take one story and ask "can you give me an example?" until the rules are clear.

BDD is not the tool. Writing tests in Gherkin without the conversation gives you slower, wordier tests and none of the benefit. If only developers ever read the feature files, plain pytest is usually the better choice.

Gherkin in five keywords

Gherkin is the language the scenarios are written in. Cucumber made it popular, and pytest-bdd, Behave, SpecFlow and many others read it. A feature file looks like this:

Feature: Checkout pricing

  Scenario: Delivery is charged on a small order
    Given my basket has food worth ₹300
    When I check out
    Then delivery costs ₹40
    And I pay ₹340

Each line is a step. In the test code, every step is matched to a Python function, its step definition.

Step 1: Set up

You need Python 3.10 or newer:

mkdir foodnow-bdd && cd foodnow-bdd
python -m venv .venv && source .venv/bin/activate
pip install pytest==9.2.0 pytest-bdd==9.0.1
mkdir -p foodnow tests/features
touch foodnow/__init__.py
printf '[pytest]\npythonpath = .\n' > pytest.ini

pytest-bdd installs the parse library (1.22.4 here), which we use in Step 4.

Step 2: The pricing code, with two bugs

The rules from the product owner: delivery costs ₹40, and it's free from ₹500. The code FIRST50 takes 50% off the food, up to ₹100. Save this as foodnow/pricing.py:

"""FoodNow checkout pricing. All money is in paise (₹1 = 100 paise)."""
from dataclasses import dataclass

DELIVERY_FEE = 4000          # ₹40
FREE_DELIVERY_FROM = 50000   # ₹500
PROMOS = {"FIRST50": {"percent": 50, "cap": 10000}}  # 50% off, up to ₹100


class InvalidPromo(Exception):
    pass


@dataclass
class Bill:
    food: int
    discount: int
    delivery: int

    @property
    def total(self) -> int:
        return self.food - self.discount + self.delivery


def price_order(food: int, promo: str | None = None) -> Bill:
    discount = 0
    if promo is not None:
        if promo not in PROMOS:
            raise InvalidPromo(f"{promo} is not a valid code")
        rule = PROMOS[promo]
        discount = food * rule["percent"] // 100
    delivery = 0 if food > FREE_DELIVERY_FROM else DELIVERY_FEE
    return Bill(food, discount, delivery)

Two bugs are planted: > should be >=, and the ₹100 cap is never applied. Both are the kind of thing that happens when a rule is read quickly.

Step 3: The scenarios

These are the examples the three amigos agreed on. Save them as tests/features/checkout_pricing.feature:

Feature: Checkout pricing
  As a hungry customer
  I want to know exactly what I'll pay before I pay
  So that there are no surprises at the door

  Rules agreed with the product owner:
  - Delivery costs ₹40. It is free when the food comes to ₹500 or more.
  - The code FIRST50 takes 50% off the food, up to ₹100.

  Scenario: Delivery is charged on a small order
    Given my basket has food worth ₹300
    When I check out
    Then delivery costs ₹40
    And I pay ₹340

  Scenario Outline: Free delivery from ₹500
    Given my basket has food worth ₹<food>
    When I check out
    Then delivery costs ₹<delivery>

    Examples:
      | food | delivery |
      | 499  | 40       |
      | 500  | 0        |
      | 501  | 0        |

  Scenario: FIRST50 takes half off a small order
    Given my basket has food worth ₹120
    When I check out with the code "FIRST50"
    Then the discount is ₹60
    And I pay ₹100

  Scenario: FIRST50 is capped at ₹100
    Given my basket has food worth ₹400
    When I check out with the code "FIRST50"
    Then the discount is ₹100
    And I pay ₹340

  Scenario: An unknown code is refused
    Given my basket has food worth ₹300
    When I check out with the code "FREEFOOD"
    Then I see the message "FREEFOOD is not a valid code"

The text under Feature: is free text. pytest-bdd ignores it, but people read it, so it's a good place for the rules. The examples table tests both sides of ₹500 and the value itself, the boundary values where bugs live. A ₹400 basket is where the 50% discount (₹200) is more than the cap, so that's the example that checks the cap.

Notice what isn't in the scenarios: buttons, pages and clicks. "When I check out" says what the customer does, not how. That keeps the scenarios readable, and they don't break when the screen changes.

Step 4: The step definitions

Each step line needs a Python function. Save this as tests/test_checkout_pricing.py:

from parse import with_pattern
from pytest_bdd import given, parsers, scenarios, then, when

from foodnow.pricing import InvalidPromo, price_order

scenarios("features/checkout_pricing.feature")


@with_pattern(r"\d[\d,]*")
def rupees(text: str) -> int:
    """'₹1,200' in a scenario becomes 120000 paise in the test."""
    return int(text.replace(",", "")) * 100


def step(text):
    return parsers.parse(text, extra_types={"Rupees": rupees})


@given(step("my basket has food worth ₹{food:Rupees}"), target_fixture="basket")
def basket(food):
    return {"food": food}


@when("I check out", target_fixture="result")
def check_out(basket):
    return price_order(basket["food"])


@when(step('I check out with the code "{code}"'), target_fixture="result")
def check_out_with_code(basket, code):
    try:
        return price_order(basket["food"], promo=code)
    except InvalidPromo as e:
        return e


@then(step("delivery costs ₹{amount:Rupees}"))
def delivery_costs(result, amount):
    assert result.delivery == amount


@then(step("the discount is ₹{amount:Rupees}"))
def discount_is(result, amount):
    assert result.discount == amount


@then(step("I pay ₹{amount:Rupees}"))
def i_pay(result, amount):
    assert result.total == amount


@then(step('I see the message "{message}"'))
def i_see(result, message):
    assert isinstance(result, InvalidPromo)
    assert str(result) == message

How it fits together:

• scenarios(...) turns every scenario in the feature file into a pytest test.
• {food:Rupees} pulls the amount out of the step text and converts it with rupees(). The scenarios talk in rupees, as people do. The code works in paise. Step 6 explains why this is a custom type rather than the usual {food:d}.
• target_fixture="basket" makes the step's return value available as a fixture called basket, which later steps ask for. That's how a Given passes its data to the When, and the When to the Then.
• One step definition serves every scenario that uses that line. "I pay ₹{amount}" is written once and used in four scenarios.

Step 5: Run it

--gherkin-terminal-reporter prints the scenarios instead of test names, and -vv adds the steps:

$ pytest --gherkin-terminal-reporter -vv --tb=no -k "capped or 500-0"
collecting ... collected 7 items / 5 deselected / 2 selected

tests/test_checkout_pricing.py::test_free_delivery_from_500[500-0]
Feature: Checkout pricing
    Scenario Outline: Free delivery from ₹500
        Given my basket has food worth ₹500
        When I check out
        Then delivery costs ₹0
    FAILED


tests/test_checkout_pricing.py::test_first50_is_capped_at_100
Feature: Checkout pricing
    Scenario: FIRST50 is capped at ₹100
        Given my basket has food worth ₹400
        When I check out with the code "FIRST50"
        Then the discount is ₹100
        And I pay ₹340
    FAILED


=========================== short test summary info ============================
FAILED tests/test_checkout_pricing.py::test_free_delivery_from_500[500-0] - assert 4000 == 0
 +  where 4000 = Bill(food=50000, discount=0, delivery=4000).delivery
FAILED tests/test_checkout_pricing.py::test_first50_is_capped_at_100 - assert 20000 == 10000
 +  where 20000 = Bill(food=40000, discount=20000, delivery=4000).discount
======================= 2 failed, 5 deselected in 0.02s ========================

Without -k, the full run is 2 failed, 5 passed. Both planted bugs, and the output is something the product owner can read: "₹500 of food, delivery should cost ₹0, it failed." The ₹499 and ₹501 rows passed, so only the exact boundary was wrong.

The fix:

-        discount = food * rule["percent"] // 100
-    delivery = 0 if food > FREE_DELIVERY_FROM else DELIVERY_FEE
+        discount = min(food * rule["percent"] // 100, rule["cap"])
+    delivery = 0 if food >= FREE_DELIVERY_FROM else DELIVERY_FEE

$ pytest -q
.......                                                                  [100%]
7 passed in 0.03s

Step 6: The product owner writes "₹1,200"

Scenarios are meant to be written and read by people who don't code. So the product owner adds one:

  Scenario: A big order gets free delivery
    Given my basket has food worth ₹1,200
    When I check out
    Then delivery costs ₹0
    And I pay ₹1,200

Our first step definitions used the usual {food:d}, which matches digits only. The comma doesn't match:

E   pytest_bdd.exceptions.StepDefinitionNotFoundError: Step definition is not found: Given "my basket has food worth ₹1,200". Line 45 in scenario "A big order gets free delivery" in the feature ".../tests/features/checkout_pricing.feature"
1 failed, 7 passed in 0.02s

This failure is loud, which is good. The wrong fix is to ask the product owner to write "1200". The right fix is to make the steps accept the way people write money. That's the Rupees type in Step 4: it allows commas and converts to paise in one place. With it, all 8 pass. When a scenario and a step definition disagree, the scenario is usually the one that's right, because it's in the language of the people who own the rule.

Step 7: The question that wasn't a bug

Back to the three amigos. Picture this exchange: the tester asks, "₹520 of food with FIRST50. The discount is ₹100, so the customer pays ₹420 for the food. Is delivery free?"

The developer says yes: the code checks the food total, ₹520, before the discount. The product owner says no: free delivery is for orders where the customer pays ₹500 or more for food. Both had read the same rule. Neither was careless. The rule was ambiguous, and nobody noticed until someone asked for an example.

The product owner's answer becomes a scenario:

  Scenario: A discount that takes the food below ₹500
    Given my basket has food worth ₹520
    When I check out with the code "FIRST50"
    Then the discount is ₹100
    And delivery costs ₹40
    And I pay ₹460

$ pytest --gherkin-terminal-reporter -vv --tb=no -k below
collecting ... collected 9 items / 8 deselected / 1 selected

tests/test_checkout_pricing.py::test_a_discount_that_takes_the_food_below_500
Feature: Checkout pricing
    Scenario: A discount that takes the food below ₹500
        Given my basket has food worth ₹520
        When I check out with the code "FIRST50"
        Then the discount is ₹100
        And delivery costs ₹40
        And I pay ₹460
    FAILED


=========================== short test summary info ============================
FAILED tests/test_checkout_pricing.py::test_a_discount_that_takes_the_food_below_500 - assert 0 == 4000
 +  where 0 = Bill(food=52000, discount=10000, delivery=0).delivery
======================= 1 failed, 8 deselected in 0.01s ========================

The code did what the developer understood. It wasn't a bug in the code. It was a gap in the rule, and BDD is mainly a tool for finding those. One line changes:

-    delivery = 0 if food >= FREE_DELIVERY_FROM else DELIVERY_FEE
+    delivery = 0 if food - discount >= FREE_DELIVERY_FROM else DELIVERY_FEE

And the rule at the top of the feature file is updated, so the next person doesn't have the same argument:

  - Delivery costs ₹40. It is free when the food, after any discount,
    comes to ₹500 or more.

$ pytest --gherkin-terminal-reporter -v --cucumberjson=cucumber.json
...
tests/test_checkout_pricing.py::test_a_big_order_gets_free_delivery
Feature: Checkout pricing
    Scenario: A big order gets free delivery PASSED

tests/test_checkout_pricing.py::test_a_discount_that_takes_the_food_below_500
Feature: Checkout pricing
    Scenario: A discount that takes the food below ₹500 PASSED

- generated json file: ~/foodnow-bdd/cucumber.json -
============================== 9 passed in 0.02s ===============================

--cucumberjson writes the results in Cucumber's JSON format, with each step and its status. Many reporting tools can read it and show the scenarios as a readable report, sometimes called living documentation: a description of the system that's checked against the code on every run, so it can't quietly go out of date.

Step 8: The scenario that never ran

pytest-bdd has two ways to turn scenarios into tests. scenarios("file.feature"), used above, binds every scenario in the file. The other way binds them one at a time:

from pytest_bdd import scenario

FEATURE = "features/checkout_pricing.feature"


@scenario(FEATURE, "Delivery is charged on a small order")
def test_small_order():
    pass


@scenario(FEATURE, "FIRST50 is capped at ₹100")
def test_first50_cap():
    pass

Teams sometimes do this to give tests their own names. We copied the feature file, with all seven scenarios, next to a test file bound this way, and ran it on the pricing code from Step 5, before the after-discount fix:

$ pytest trap -v
collecting ... collected 2 items

trap/test_pricing_bound_one_by_one.py::test_small_order PASSED           [ 50%]
trap/test_pricing_bound_one_by_one.py::test_first50_cap PASSED           [100%]

============================== 2 passed in 0.02s ===============================

Green. The product owner's new scenario for ₹520 is in the feature file, but nothing binds it, so it never runs, and there's no warning. The same code with scenarios() gives 1 failed, 8 passed. For people who don't read the Python, this is the worst failure possible: they wrote the rule down, they see a green build, and they believe it's checked. Use scenarios(), and if you must bind one by one, check that the number of tests matches the number of scenarios.

When BDD is worth it

• Worth it: rules that business people own and argue about, like pricing, eligibility, discounts, refunds and permissions. When the scenarios are read and changed by people who don't code.
• Not worth it: technical details nobody outside the team reads, like a retry backoff or a parser. Plain pytest is shorter and easier to change.
• Best value: before the code. Scenarios written afterwards, to describe what the code already does, find the planted kind of bug, but not the Step 7 kind.

Common mistakes

• Scenarios that describe clicks. "When I click the button with id pay-btn" ties the scenario to the screen and nobody outside the team wants to read it. Describe what the user does: "When I check out".
• Gherkin without the conversation. If only developers write and read the scenarios, you have slower tests, not BDD.
• Too many steps per scenario. If a scenario needs fifteen steps, it's testing several things. One behaviour per scenario.
• Step definitions that are too strict. A step that only accepts "1200" makes people write for the tool. Accept the way people write, as in Step 6.
• Binding scenarios one by one. New scenarios don't run, and nothing warns you. Use scenarios().
• Changing the code without updating the rule. When a scenario settles a question, update the rule text in the feature file too, as in Step 7.

What's next

In Part 9 of Testing in Practice we look at user acceptance testing (UAT): how the people who'll use the software decide whether it's ready, and how to plan, run and record it without it turning into a rubber stamp.

ProvaLab's pytest plugin works with pytest-bdd tests like any other pytest test, and sends each scenario's result to the ProvaLab test management system with its history across runs. You can link a scenario to the requirement it checks, so a product owner can see which agreed rules are covered and which are passing. See it in the two-minute product tour.

Both teams tested. What nobody tested was the agreement between them: the requests the app sends and the answers it needs back. That agreement is a contract, and contract testing checks it without running the app and the API together.

This post shows how contract testing works with Pact, the most widely used tool for it. Then it runs the story above for real, with FoodNow's app code and orders API. Every command and line of output below was run while writing this post.

Why not just test them together?

You can start the app and the real API in one environment and run tests across both. That's an end-to-end or integration test, and you should have a few. But as the only check on the agreement, they're a poor fit:

• They're slow and flaky. Every service, database and network link must be up at once.
• They tell you something broke, late. Often after both changes were merged.
• They don't scale. With ten apps and services, you need one environment where the right version of every one runs together.

And mocks don't solve it. In Part 5, a mock of the payment provider happily accepted calls the real provider would have refused. A mock only knows what you told it.

How Pact works

Contract testing splits the check in two. Each half runs in one team's normal test suite, with nothing else running:

1. The consumer test. The app team writes tests for their API client, against a mock server that Pact runs. Each test says "when the app sends this request, it needs this response". Pact checks that the app really sends that request, and answers with that response.
2. The pact file. When the tests pass, Pact writes every request and response to a JSON file, the contract.
3. The provider verification. The API team runs Pact's verifier against their real API. It replays every request in the pact file and checks that the real responses match.

Two words you'll see: the consumer is the side that calls (the app), and the provider is the side that answers (the orders API). A provider state is the situation a request needs, such as "order 7 exists". The API team writes a small function for each state, to set it up before the request is replayed.

This is called consumer-driven contract testing: the contract holds only what the consumer actually uses. If the API sends 30 fields and the app reads 4, only those 4 are in the contract, and the API team can change the other 26 freely.

Step 1: Set up

You need Python 3.10 or newer:

mkdir foodnow-contract && cd foodnow-contract
python -m venv .venv && source .venv/bin/activate
pip install pact-python==4.2.0 pytest==9.2.0 fastapi==0.162.4 uvicorn==0.54.0 requests==2.34.2 httpx==0.28.1
mkdir -p app_client orders_api tests/consumer tests/provider
touch app_client/__init__.py orders_api/__init__.py tests/__init__.py tests/consumer/__init__.py tests/provider/__init__.py
printf '[pytest]\npythonpath = .\n' > pytest.ini
export PACT_DO_NOT_TRACK=true

pact-python 4 runs Pact's core, written in Rust, inside Python, so there is nothing else to install. By default, Pact sends anonymous usage statistics (the Pact version and operating system); PACT_DO_NOT_TRACK=true turns that off.

In a real project the two halves live in two repositories, owned by two teams. Here they share one folder so you can run everything.

Step 2: The app's API client

This is the part of the FoodNow app that talks to the orders API. It reads an order, and it places a new one with a tip. Save it as app_client/client.py:

"""The FoodNow app's code for talking to the orders API."""
from dataclasses import dataclass

import requests


@dataclass
class Order:
    id: int
    status: str
    total_paise: int
    tip_paise: int


class OrdersClient:
    def __init__(self, base_url: str):
        self.base_url = base_url

    def get_order(self, order_id: int) -> Order | None:
        r = requests.get(f"{self.base_url}/orders/{order_id}", timeout=5)
        if r.status_code == 404:
            return None
        r.raise_for_status()
        d = r.json()
        return Order(d["id"], d["status"], d["total_paise"], d["tip_paise"])

    def place_order(self, items: list[dict], tip_paise: int) -> Order:
        r = requests.post(
            f"{self.base_url}/orders",
            json={"items": items, "tip_paise": tip_paise},
            timeout=5,
        )
        r.raise_for_status()
        d = r.json()
        return Order(d["id"], d["status"], d["total_paise"], d["tip_paise"])

Money is in paise (1 rupee = 100 paise) as a whole number, so there are no rounding errors.

Step 3: The consumer test writes the contract

Save this as tests/consumer/test_orders_contract.py. Each test describes one interaction, starts Pact's mock server, and calls the real client code against it:

from pathlib import Path

import pytest
from pact import Pact, match

from app_client.client import OrdersClient

PACTS = Path(__file__).parents[2] / "pacts"


@pytest.fixture
def pact():
    pact = Pact("foodnow-app", "orders-api").with_specification("V4")
    yield pact
    pact.write_file(PACTS)  # adds this test's interaction to the file


def test_get_order(pact):
    (
        pact.upon_receiving("a request for an order")
        .given("order 7 exists")
        .with_request("GET", "/orders/7")
        .will_respond_with(200)
        .with_body(
            {
                "id": match.int(7),
                "status": match.str("preparing"),
                "total_paise": match.int(40000),
                "tip_paise": match.int(4000),
            }
        )
    )
    with pact.serve() as srv:
        order = OrdersClient(str(srv.url)).get_order(7)
    assert order.total_paise == 40000
    assert order.tip_paise == 4000


def test_missing_order(pact):
    (
        pact.upon_receiving("a request for an order that doesn't exist")
        .given("order 999 does not exist")
        .with_request("GET", "/orders/999")
        .will_respond_with(404)
    )
    with pact.serve() as srv:
        assert OrdersClient(str(srv.url)).get_order(999) is None


def test_place_order(pact):
    (
        pact.upon_receiving("a new order with a tip")
        .with_request("POST", "/orders")
        .with_body({"items": [{"name": "paneer wrap", "qty": 2}], "tip_paise": 5000})
        .will_respond_with(201)
        .with_body(
            {
                "id": match.int(1),
                "status": match.str("placed"),
                "total_paise": 41000,
                "tip_paise": 5000,
            }
        )
    )
    with pact.serve() as srv:
        order = OrdersClient(str(srv.url)).place_order(
            [{"name": "paneer wrap", "qty": 2}], tip_paise=5000
        )
    assert order.total_paise == 41000

Two kinds of expected value are used on purpose:

• Matchers, like match.int(7), mean "any whole number; 7 is just an example". Use them where the exact value doesn't matter. The order id depends on what's in the API's database, so the contract shouldn't fix it.
• Exact values, like "total_paise": 41000, mean "exactly this". Two paneer wraps at ₹180 plus a ₹50 tip is ₹410, whatever is in the database. Here the value is the point. Step 7 shows why that choice matters.

The fixture is created fresh for each test, because a pact can't be changed after its mock server has started. Add tests/consumer/conftest.py to start every run with an empty pact folder:

import shutil
from pathlib import Path


def pytest_sessionstart(session):
    shutil.rmtree(Path(__file__).parents[2] / "pacts", ignore_errors=True)

Run it:

$ pytest tests/consumer -q
...                                                                      [100%]
3 passed in 0.07s

The contract is now in pacts/foodnow-app-orders-api.json. This is the part for reading an order:

{
  "description": "a request for an order",
  "providerStates": [{ "name": "order 7 exists" }],
  "request": { "method": "GET", "path": "/orders/7" },
  "response": {
    "status": 200,
    "body": {
      "content": { "id": 7, "status": "preparing", "tip_paise": 4000, "total_paise": 40000 }
    },
    "matchingRules": {
      "body": {
        "$.id":          { "combine": "AND", "matchers": [{ "match": "integer" }] },
        "$.status":      { "combine": "AND", "matchers": [{ "match": "type" }] },
        "$.tip_paise":   { "combine": "AND", "matchers": [{ "match": "integer" }] },
        "$.total_paise": { "combine": "AND", "matchers": [{ "match": "integer" }] }
      }
    }
  }
}

(We removed the headers and some fields Pact adds, to keep it short.) This file is what the app team hands to the API team.

Step 4: The provider verifies the contract

Here's version 1 of the orders API, in orders_api/app.py. It's a small FastAPI app that keeps orders in memory:

"""FoodNow orders API, version 1."""
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel

app = FastAPI()

MENU = {"paneer wrap": 18000, "masala chai": 4000}
ORDERS: dict[int, dict] = {}


class Item(BaseModel):
    name: str
    qty: int


class NewOrder(BaseModel):
    items: list[Item]
    tip_paise: int = 0


@app.get("/orders/{order_id}")
def get_order(order_id: int):
    if order_id not in ORDERS:
        raise HTTPException(404, "order not found")
    return ORDERS[order_id]


@app.post("/orders", status_code=201)
def place_order(body: NewOrder):
    food = sum(MENU[i.name] * i.qty for i in body.items)
    order_id = max(ORDERS, default=0) + 1
    return save_order(order_id, food, body.tip_paise)


def save_order(order_id: int, food_paise: int, tip_paise: int, status: str = "placed"):
    ORDERS[order_id] = {
        "id": order_id,
        "status": status,
        "total_paise": food_paise + tip_paise,
        "tip_paise": tip_paise,
    }
    return ORDERS[order_id]

The verification test starts the real API on a local port, tells Pact how to set up each provider state, and runs the verifier. Save it as tests/provider/test_verify_contract.py:

import threading
import time
from pathlib import Path

import pytest
import uvicorn
from pact import Verifier

from orders_api import app as api

PACTS = Path(__file__).parents[2] / "pacts"


@pytest.fixture(scope="module")
def server():
    config = uvicorn.Config(api.app, port=8123, log_level="warning")
    srv = uvicorn.Server(config)
    threading.Thread(target=srv.run, daemon=True).start()
    while not srv.started:
        time.sleep(0.05)
    yield "http://localhost:8123"
    srv.should_exit = True


def order_7_exists(action, parameters=None):
    if action == "setup":
        api.ORDERS[7] = {"id": 7, "status": "preparing",
                         "total_paise": 40000, "tip_paise": 4000}


def order_999_missing(action, parameters=None):
    api.ORDERS.pop(999, None)


def reset(action, parameters=None):
    if action == "setup":
        api.ORDERS.clear()


def test_orders_api_keeps_the_contract(server):
    verifier = (
        Verifier("orders-api")
        .add_transport(url=server)
        .set_coloured_output(enabled=False)
        .add_source(PACTS)
        .state_handler(
            {
                "": reset,  # for interactions with no given state
                "order 7 exists": order_7_exists,
                "order 999 does not exist": order_999_missing,
            },
            teardown=True,
        )
    )
    verifier.verify()

Run it with -s to see the verifier's report:

$ pytest tests/provider/test_verify_contract.py -q -s
Verifying a pact between foodnow-app and orders-api

  a new order with a tip (0s loading, 15ms verification)
    returns a response which
      has status code 201 (OK)
      includes headers
        "Content-Type" with value "application/json" (OK)
      has a matching body (OK)

  a request for an order (0s loading, 8ms verification)
     Given order 7 exists
    returns a response which
      has status code 200 (OK)
      includes headers
        "Content-Type" with value "application/json" (OK)
      has a matching body (OK)

  a request for an order that doesn't exist (0s loading, 6ms verification)
     Given order 999 does not exist
    returns a response which
      has status code 404 (OK)
      has a matching body (OK)

.
1 passed in 0.79s

The app and the API agree. Look at order_7_exists, though: it writes an order straight into the API's storage. Keep that in mind.

Step 5: The API team changes how money is sent

Now the API team ships version 2. Money becomes an amount plus a currency, and the request field tip_paise is renamed to tip. These are the lines that change:

class NewOrder(BaseModel):
    items: list[Item]
    tip: int = 0  # paise


def money(paise: int) -> dict:
    return {"amount_paise": paise, "currency": "INR"}

# in place_order:
    return save_order(order_id, food, body.tip)

# in save_order:
    ORDERS[order_id] = {
        "id": order_id,
        "status": status,
        "total": money(food_paise + tip_paise),
        "tip": money(tip_paise),
    }

They also update their own tests, in tests/provider/test_api.py:

"""The orders team's own tests, updated for version 2."""
from fastapi.testclient import TestClient

from orders_api import app as api

client = TestClient(api.app)


def test_place_order_adds_the_tip():
    r = client.post("/orders", json={"items": [{"name": "paneer wrap", "qty": 2}], "tip": 5000})
    assert r.status_code == 201
    assert r.json()["total"] == {"amount_paise": 41000, "currency": "INR"}


def test_missing_order_is_404():
    assert client.get("/orders/999").status_code == 404

$ pytest tests/provider/test_api.py -q
..                                                                       [100%]
2 passed, 1 warning in 0.25s

Green. (The warning is a deprecation notice from Starlette's test client, not from this code.) The app team's tests are green too, because they haven't changed anything. Now the contract:

$ pytest tests/provider/test_verify_contract.py -q
...
Failures:

1) Verifying a pact between foodnow-app and orders-api - a new order with a tip
    1.1) has a matching body
           $ -> Actual map is missing the following keys: tip_paise, total_paise
    {
  "id": 1,
  "status": "placed",
    -  "tip_paise": 5000,
  "total_paise": 41000
    +  "tip": {
    "amount_paise": 0,
    "currency": "INR"
  },
  "total": {
    "amount_paise": 36000,
    "currency": "INR"
  }
    }

There were 1 pact failures
FAILED tests/provider/test_verify_contract.py::test_orders_api_keeps_the_contract
1 failed in 0.86s

Pact caught the change before release, and the diff shows a second problem nobody planned for. Read the numbers: the tip is 0 and the total is ₹360, not ₹410. The app still sends tip_paise. Version 2 only knows tip, and Pydantic ignores fields it doesn't know, so the tip is silently dropped. No error, no 422. The customer is charged less, and the delivery rider loses the tip.

But only one interaction failed. Reading an order passed, although version 2 sends total too. Why?

Step 6: The state handler that hid a failure

Back to order_7_exists from Step 4:

api.ORDERS[7] = {"id": 7, "status": "preparing",
                 "total_paise": 40000, "tip_paise": 4000}

It writes the order straight into storage, in the old shape. The GET endpoint returns whatever is stored, so the verifier got back exactly the version 1 response the test had written in. The test checked the state handler, not the API.

The fix is to create test data through the provider's own code, the same way real data is created. Both versions of the API have a save_order function, so:

def order_7_exists(action, parameters=None):
    if action == "setup":
        api.save_order(7, food_paise=36000, tip_paise=4000, status="preparing")

With the fixed state handler, version 1 still passes. Version 2 now fails both interactions:

2) Verifying a pact between foodnow-app and orders-api Given order 7 exists - a request for an order
    2.1) has a matching body
           $ -> Actual map is missing the following keys: tip_paise, total_paise
    {
  "id": 7,
  "status": "preparing",
    -  "tip_paise": 4000,
  "total_paise": 40000
    +  "tip": {
    "amount_paise": 4000,
    "currency": "INR"
  },
  "total": {
    "amount_paise": 40000,
    "currency": "INR"
  }
    }

There were 2 pact failures

This is the most common way contract tests lie. If a state handler builds the response data by hand, the contract only proves that the handler agrees with the contract.

Step 7: Change the API without breaking the app

The API team can still move to the new money format. They just can't remove the old fields while app versions that read them are still on customers' phones. The usual pattern is expand, then contract: add the new fields next to the old ones, wait until no app version uses the old ones, then remove them.

Their first try puts the old response fields back:

    ORDERS[order_id] = {
        "id": order_id,
        "status": status,
        "total": money(food_paise + tip_paise),
        "tip": money(tip_paise),
        # kept until every app version has moved to "total" and "tip"
        "total_paise": food_paise + tip_paise,
        "tip_paise": tip_paise,
    }

1) Verifying a pact between foodnow-app and orders-api - a new order with a tip
    1.1) has a matching body
           $.tip_paise -> Expected 0 (Integer) to be equal to 5000 (Integer)
           $.total_paise -> Expected 36000 (Integer) to be equal to 41000 (Integer)

There were 1 pact failures

The fields are back, but the tip is still being dropped: the request side was missed. This failure exists only because the consumer test used exact values for the total and tip in Step 3. To check, we changed those two to match.int(41000) and match.int(5000), regenerated the pact and ran the verification again:

1 passed in 0.79s

With type matchers, Pact passed an API that drops every tip, because 0 and 36,000 are whole numbers too. Matchers check the shape. They don't check that the numbers are right. Use exact values where the value comes straight from the request, and add your own functional tests for the rest. Contract tests are not a replacement for those.

We put the exact values back. The second fix accepts the old request field too:

class NewOrder(BaseModel):
    items: list[Item]
    tip: int = 0  # paise
    tip_paise: int | None = None  # old name, still sent by older app versions

# in place_order:
    tip = body.tip if body.tip_paise is None else body.tip_paise
    return save_order(order_id, food, tip)

The whole suite, consumer and provider:

$ pytest tests -v
platform linux -- Python 3.11.17, pytest-9.2.0, pluggy-1.7.0
rootdir: ~/foodnow-contract
configfile: pytest.ini
plugins: pact-python-4.2.0, anyio-4.16.0
collected 6 items

tests/consumer/test_orders_contract.py::test_get_order PASSED            [ 16%]
tests/consumer/test_orders_contract.py::test_missing_order PASSED        [ 33%]
tests/consumer/test_orders_contract.py::test_place_order PASSED          [ 50%]
tests/provider/test_api.py::test_place_order_adds_the_tip PASSED         [ 66%]
tests/provider/test_api.py::test_missing_order_is_404 PASSED             [ 83%]
tests/provider/test_verify_contract.py::test_orders_api_keeps_the_contract PASSED [100%]

========================= 6 passed, 1 warning in 0.87s =========================

Old app versions keep working, and new versions can start reading total and tip. When every app version still in use has moved to them, no contract mentions total_paise any more, and the API team can remove it.

Step 8: The contract checks the app too

The contract works in both directions. Suppose the app team is the one who renames tip_paise to tip, in place_order in the client. Their contract test fails straight away, in their own suite:

E           requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://localhost:37013/orders
...
E               pact.error.MismatchesError: Mismatches:
E                 (1) Request mismatch: POST /orders
E                   (1) Body mismatch: Expected a Map with keys [items, tip_paise] but received one with keys [items, tip]
1 failed, 2 passed in 0.11s

Pact's mock server answers a request that doesn't match the contract with a 500, then names the mismatch. So the app can't quietly start sending something the contract doesn't describe. If the change is meant, the app team updates the contract, and the API team's verification shows whether the API can handle it.

Sharing the contract between teams

In this post the pact file sits in a folder both halves can read. Real teams need a way to pass it from the app's CI to the API's CI. The usual answer is a Pact Broker, an open-source server that stores pact files and verification results by version. (We didn't run a broker for this post.) With a broker:

• The app's CI publishes the pact after its tests pass.
• The API's CI fetches the pacts for all the app versions in production and verifies against them.
• Before deploying either side, a command called can-i-deploy asks the broker whether this version has been verified against the versions it will talk to. If not, the deploy stops.

PactFlow is a hosted broker with extra features, run by SmartBear.

Where contract tests fit

Contract tests replace most of the end-to-end tests that only exist to check the "wiring" between services. Keep a few end-to-end tests for the journeys that matter most.

Common mistakes

• State handlers that build the response by hand. Step 6 passed a broken API this way. Create test data through the provider's own code or API.
• Matchers everywhere. They check shape, not values. Step 7 shows a contract with only type matchers passing an API that drops every tip.
• Exact values everywhere. The opposite problem: the contract breaks whenever test data changes, people stop trusting it, and they turn it off. Use exact values only where the value follows from the request.
• Putting the whole API in the contract. Include only what the consumer actually uses. A field the app never reads shouldn't stop the API team from changing it.
• Removing old fields as soon as the new ones ship. Older app versions stay on phones for months. Expand first, contract later.
• Treating contract tests as functional tests. They check the agreement, not the business rules.

What's next

In Part 8 of Testing in Practice we look at behaviour-driven development (BDD): writing tests as Given/When/Then scenarios that product owners can read, and running them with pytest-bdd.

ProvaLab doesn't replace Pact, but both halves here are ordinary pytest tests. ProvaLab's pytest plugin sends every result to the ProvaLab test management system, so a failed contract verification shows up in the API team's test run, with the test's history across runs, and you can link the test to the requirement it protects. See it in the two-minute product tour.

This post writes a first Cypress test for the same page. This time the total comes from a price API, which lets us use Cypress's best feature, cy.intercept, to make the network fail and answer out of order. The page has three bugs, and two of them are only reachable that way. And one of our own tests passed on the buggy page. Every command and line of output below was run while writing this post, with Cypress 15.21.1.

How Cypress is different

• The test runs in the browser, in the same tab as your app. It sees changes the moment they happen, and you can step back through every command in Cypress's log.
• Commands queue and retry. cy.get("#total").should("have.text", "₹220") keeps looking for 4 seconds by default, until the text matches.
• Cypress sits on the network. Its proxy sees every request the app makes, and cy.intercept can watch, delay, change or replace any of them.
• The costs: tests are JavaScript or TypeScript only, each test runs in one browser tab, and Safari is only available through an experimental WebKit option.

Step 1: Set up

You need Node.js 20 or newer:

mkdir foodnow-cypress && cd foodnow-cypress
npm init -y
npm install --save-dev cypress@15.21.1
mkdir -p site cypress/e2e

The first install downloads the Cypress app (a few hundred MB) into your home folder. npx cypress verify checks that it can run:

$ npx cypress verify
✔  Verified Cypress!       /home/test/.cache/Cypress/15.21.1/Cypress

Step 2: The page and a small server

The order page is the one from Part 11, with one change: the total now comes from a price API. Here's the part of site/index.html that changed:

  function render() {
    // ...draws the basket lines, each with a Remove button:
    //   row.querySelector("button").onclick = () => { cart.splice(i + 1, 1); render(); };

    // The total comes from the price service (prices, offers, delivery).
    document.getElementById("total").textContent = "…";
    const names = cart.map(x => x.name).join(",");
    fetch(`/api/price?items=${encodeURIComponent(names)}`)
      .then(r => r.json())
      .then(data => { document.getElementById("total").textContent = `₹${data.total}`; });
  }

Yes, the Remove bug from Part 11 is still there. We left it in so you can compare how the two tools report it.

And server.js, which serves the page and a price API that answers after 800 ms:

// A tiny FoodNow server: the order page, and a price API that answers after 800 ms.
const http = require("http");
const fs = require("fs");
const path = require("path");

const PRICES = { "Paneer wrap": 180, "Masala chai": 40, "Veg biryani": 260 };
const DELIVERY = 40, FREE_FROM = 500;

http.createServer((req, res) => {
  const url = new URL(req.url, "http://localhost");
  if (url.pathname === "/api/price") {
    const items = (url.searchParams.get("items") || "").split(",").filter(Boolean);
    const food = items.reduce((s, name) => s + PRICES[name], 0);
    const total = food === 0 ? 0 : food + (food >= FREE_FROM ? 0 : DELIVERY);
    setTimeout(() => {
      res.writeHead(200, { "Content-Type": "application/json" });
      res.end(JSON.stringify({ food, total }));
    }, 800);
    return;
  }
  if (url.pathname === "/") {
    res.writeHead(200, { "Content-Type": "text/html; charset=utf-8" });
    res.end(fs.readFileSync(path.join(__dirname, "site", "index.html")));
    return;
  }
  res.writeHead(404);
  res.end();
}).listen(process.env.PORT || 8090, () => console.log("FoodNow on http://localhost:" + (process.env.PORT || 8090)));

Start it in a second terminal with node server.js, and tell Cypress where it is in cypress.config.js:

const { defineConfig } = require("cypress");

module.exports = defineConfig({
  e2e: {
    baseUrl: "http://localhost:8090",
    supportFile: false,
    viewportWidth: 1280,
    viewportHeight: 800,
  },
});

Step 3: The first test

Save this as cypress/e2e/first.cy.js:

describe("FoodNow order", () => {
  it("adds a wrap", () => {
    cy.visit("/");
    cy.get("[data-testid=add-wrap]").click();
    cy.get("#total").should("have.text", "₹220"); // ₹180 + ₹40 delivery
  });
});

Run it headless with npx cypress run, or open the interactive runner with npx cypress open to watch it:

$ npx cypress run --spec cypress/e2e/first.cy.js
  FoodNow order
    1) adds a wrap

  0 passing (5s)
  1 failing

  1) FoodNow order
       adds a wrap:
     CypressError: Timed out retrying after 4050ms: `cy.click()` failed because this element:

`<button data-testid="add-wrap" data-name="Paneer wrap" data-price="180">Add</button>`

is being covered by another element:

`<div id="cookies">...</div>`

Fix this problem, or use {force: true} to disable error checking.

The same cookie banner that stopped Selenium. The difference is that Cypress retried for 4 seconds first, in case the banner went away by itself, and then said exactly what was in the way. It also suggests {force: true}. Don't take that suggestion here: forcing a click through a banner is something no customer can do. Close it as they would:

    cy.visit("/");
    cy.get("#accept").click(); // close the cookie banner first
    cy.get("[data-testid=add-wrap]").click();
    cy.get("#total").should("have.text", "₹220"); // ₹180 + ₹40 delivery

  FoodNow order
    ✓ adds a wrap (1020ms)

And here's the second difference from Selenium: no wait code at all. The total showed "…" for 800 ms, and should("have.text", "₹220") simply kept checking until it changed. The test took 1,020 ms, most of it waiting for the price API.

Step 4: The mistake every Cypress beginner makes

Cypress commands look like ordinary function calls, but they don't run when that line runs. They're queued and run later, one after another. So this, which looks perfectly reasonable, can't work:

  it("adds a wrap (the way that doesn't work)", () => {
    cy.visit("/");
    cy.get("#accept").click();
    cy.get("[data-testid=add-wrap]").click();
    const total = cy.get("#total").invoke("text");
    expect(total).to.equal("₹220");
  });

  1) FoodNow order
       adds a wrap (the way that doesn't work):
      AssertionError: expected { Object (specWindow, chainerId) } to equal '₹220'

total isn't the text. It's a Cypress chainer, a handle to a command that hasn't run yet. The fix that people reach for next is .then():

    cy.get("#total").invoke("text").then((text) => {
      expect(text).to.equal("₹220");
    });

  2) reading a value with .then()
      AssertionError: expected '…' to equal '₹220'

Now it gets the text, but only once, without retrying: .then() runs as soon as the command before it finishes, and the price hadn't arrived. Use .should() for anything you check, because only .should() retries. Keep .then() for when you need a value to do something with, not to assert on.

Step 5: The real tests

Save this as cypress/e2e/order.cy.js:

describe("FoodNow order", () => {
  beforeEach(() => {
    cy.visit("/");
    cy.get("#accept").click();
  });

  it("charges delivery on a small order", () => {
    cy.get("[data-testid=add-wrap]").click();
    cy.get("#total").should("have.text", "₹220");
  });

  it("gives free delivery from ₹500", () => {
    cy.get("[data-testid=add-biryani]").click();
    cy.get("[data-testid=add-biryani]").click();
    cy.get("#total").should("have.text", "₹520");
  });

  it("removes the item you remove", () => {
    cy.get("[data-testid=add-wrap]").click();
    cy.get("[data-testid=add-chai]").click();
    cy.contains("[data-testid=cart-line]", "Paneer wrap").find("button").click();
    cy.get("[data-testid=cart-line] span").should("have.length", 1).and("have.text", "Masala chai");
    cy.get("#total").should("have.text", "₹80");
  });

  it("won't place an empty order", () => {
    cy.get("#place").click();
    cy.get("#status").should("have.text", "Your basket is empty");
  });
});

cy.contains(selector, text) finds the basket line that contains "Paneer wrap", so we click its Remove button, not just the first one on the page.

Step 6: Make the network misbehave with cy.intercept

Real price services fail, and answer slowly, and answer in a different order from the one you asked in. Testing that against a real server is hard. With cy.intercept, it's a few lines. Add this to order.cy.js:

describe("when the price service misbehaves", () => {
  beforeEach(() => {
    cy.visit("/");
    cy.get("#accept").click();
  });

  it("says so when the price service is down", () => {
    cy.intercept("GET", "/api/price*", { statusCode: 500, body: { error: "price service down" } }).as("price");
    cy.get("[data-testid=add-wrap]").click();
    cy.wait("@price");
    cy.get("#total").should("contain.text", "couldn't work out your total");
  });

  it("shows the total for the latest basket when answers arrive out of order", () => {
    // Answer 1 (wrap only) arrives late; answer 2 (wrap + chai) arrives first.
    const answers = [
      { body: { food: 180, total: 220 }, delay: 1500 },
      { body: { food: 220, total: 260 }, delay: 100 },
    ];
    cy.intercept("GET", "/api/price*", (req) => req.reply(answers.shift())).as("price");
    cy.get("[data-testid=add-wrap]").click();
    cy.get("[data-testid=add-chai]").click();
    cy.wait(["@price", "@price"]); // both answers have been sent to the page
    cy.get("#total").should("have.text", "₹260");
  });
});

• The first test replaces the price API's answer with an error, as if the service were down. We expect the page to say it couldn't work out the total.
• The second plays out a real race. The customer adds a wrap, then a chai. Two price requests go out. The answer for "wrap only" is slow and arrives after the answer for "wrap and chai". The page should still show ₹260, the total for what's actually in the basket.
• .as("price") names the intercepted request, and cy.wait("@price") waits for it.

Step 7: Run it

$ npx cypress run --spec cypress/e2e/order.cy.js
  FoodNow order
    ✓ charges delivery on a small order (1020ms)
    ✓ gives free delivery from ₹500 (1048ms)
    1) removes the item you remove
    ✓ won't place an empty order (187ms)

  when the price service misbehaves
    2) says so when the price service is down
    ✓ shows the total for the latest basket when answers arrive out of order (1660ms)

  4 passing (13s)
  2 failing

Two bugs, and one test we'll come back to:

  1) FoodNow order
       removes the item you remove:
      Timed out retrying after 4000ms
      + expected - actual
      -'Paneer wrap'
      +'Masala chai'

  2) when the price service misbehaves
       says so when the price service is down:
      Timed out retrying after 4000ms
      + expected - actual
      -'₹undefined'
      +'couldn\'t work out your total'

1. Remove removes the wrong item, the same bug Selenium found: cart.splice(i + 1, 1).
2. When the price service is down, the customer sees "Total: ₹undefined". The page never checks whether the request worked. It reads the error's JSON as if it were a price, finds no total, and prints undefined. And the Place order button still works.

Cypress takes a screenshot of every failure in cypress run mode, with its command log beside the page:

Step 8: The test that passed too early

Look again at the run: the out-of-order test passed, in 1,660 ms. But the page has no protection against out-of-order answers at all: whichever answer arrives last wins. So how did it pass?

We ran exactly the same test on the same buggy page eight times. It failed six times, with -'₹220' +'₹260', and passed twice. A flaky test, on a page with a real bug.

The test does wait for both answers. We measured that: cy.wait("@price") on a reply delayed by 2,000 ms finished after 2,058 ms. But cy.wait finishes the moment the reply is sent to the page. The page updates the total a few milliseconds later. In that gap, the total still shows ₹260 from the fast answer, and should("have.text", "₹260") sees it and passes at once. Whether the test passes depends on a few milliseconds.

The deeper problem: a retrying assertion checks whether something becomes true. This test needs to check that something stays true after the late answer lands, and there's nothing to retry towards.

When there's no event that means "the page has finished reacting", a short, explained pause is the honest tool:

    cy.wait(["@price", "@price"]); // both answers have been sent to the page
    // We're checking that the total *stays* right after the late answer lands.
    // Retrying assertions can't wait for "nothing changes", so give the page a moment.
    cy.wait(300);
    cy.get("#total").should("have.text", "₹260");

With that, the test failed on the buggy page 5 times out of 5, and passed on the fixed page 5 times out of 5:

  3) when the price service misbehaves
       shows the total for the latest basket when answers arrive out of order:
      Timed out retrying after 4000ms
      + expected - actual
      -'₹220'
      +'₹260'

The customer has a wrap and a chai in the basket, and the page says ₹220, the price of the wrap alone.

Normally, cy.wait(300) with a number is a mistake, the Cypress version of time.sleep. Here it's the exception, because there's nothing else to wait for, and the comment says why. If you can, add something to the app that tests can wait for instead, such as a data-loading attribute that's removed when the last price has been applied.

Step 9: Fix the page

Three changes to render():

   const cart = [];
+  let latestRequest = 0;
...
-      row.querySelector("button").onclick = () => { cart.splice(i + 1, 1); render(); };
+      row.querySelector("button").onclick = () => { cart.splice(i, 1); render(); };
...
+    const request = ++latestRequest;
     fetch(`/api/price?items=${encodeURIComponent(names)}`)
-      .then(r => r.json())
-      .then(data => { document.getElementById("total").textContent = `₹${data.total}`; });
+      .then(r => { if (!r.ok) throw new Error(`price service: ${r.status}`); return r.json(); })
+      .then(data => {
+        if (request !== latestRequest) return; // an older answer: the basket has changed since
+        document.getElementById("total").textContent = `₹${data.total}`;
+      })
+      .catch(() => {
+        if (request !== latestRequest) return;
+        document.getElementById("total").textContent = "We couldn't work out your total. Please try again.";
+      });

• Remove the item that was clicked.
• Treat a failed request as a failure, and tell the customer.
• Number each request, and ignore any answer that isn't for the latest basket.

$ npx cypress run --spec cypress/e2e/order.cy.js
  FoodNow order
    ✓ charges delivery on a small order (1025ms)
    ✓ gives free delivery from ₹500 (1027ms)
    ✓ removes the item you remove (1093ms)
    ✓ won't place an empty order (186ms)

  when the price service misbehaves
    ✓ says so when the price service is down (187ms)
    ✓ shows the total for the latest basket when answers arrive out of order (1953ms)

  6 passing (6s)

We ran it five times on the fixed page: 6 passing every time. The two network tests that use stubs are also the fastest, because they don't wait for the real 800 ms server.

Selenium, Playwright or Cypress?

Choose Cypress when your team writes JavaScript, tests mostly one web app, and wants the friendliest runner to write and debug tests in. Choose Playwright for more browsers and languages and multi-tab flows. Keep Selenium for existing suites, real Safari, and device clouds built around WebDriver. All three found the Remove bug. What found the other two was the decision to test what happens when the network misbehaves.

Common mistakes

• Treating commands as values. const t = cy.get(...) is a chainer, not an element or its text. Chain .should() instead.
• Asserting inside .then(). It doesn't retry. Use .should().
• {force: true} to get past overlays. It clicks through things users can't. Close the overlay.
• Numeric cy.wait(ms) everywhere. Wait for an alias (cy.wait("@price")) or an assertion. Use a number only when nothing else will do, and say why in a comment.
• Testing only the happy network. cy.intercept makes errors, slow answers and out-of-order answers cheap to test. Two of our three bugs were there.
• Trusting a pass you didn't expect. Our race test passed on a page with no race protection. When a test passes and you can't say why the code is right, find out.

What's next

In Part 13 of Testing in Practice we look at test estimation: how long will testing take, how to estimate it from the work rather than guess, and how to explain the number to people who wanted it smaller.

ProvaLab's QA Infrastructure runs browser tests in the cloud with a live view while they run, and ProvaLab's plugins for Cypress, Playwright and pytest send each result to the ProvaLab test management system, with screenshots and each test's history across runs. A test that passes one run and fails the next, like the race test in Step 8, shows both results in its history instead of being lost in a CI log. See it in the two-minute product tour.

This post writes a first Selenium test in Python with pytest, against a small FoodNow order page. The first version fails twice for reasons that have nothing to do with the app. Those are the two errors almost every beginner hits, so we show them exactly. Then a proper version, with waits and a page object, finds a bug planted in the page. At the end, the same tests in Playwright, to compare. Every command and line of output below was run while writing this post.

How Selenium works

• Your test calls the Selenium library: find an element, click it, read its text.
• Selenium sends each command as an HTTP request in the WebDriver protocol to a driver: chromedriver for Chrome, geckodriver for Firefox, safaridriver for Safari.
• The driver controls the real browser, and sends back the result or an error.

You used to have to download the right driver for your browser version yourself, and update it every time the browser updated. Since Selenium 4.6, Selenium Manager does it. On our machine it found the installed Google Chrome 148 and downloaded chromedriver 148 into ~/.cache/selenium, with no setup.

One thing to remember from the diagram: every command is one request and one answer. Selenium doesn't know that your page is still loading something. If you don't tell it to wait, it won't.

Step 1: Set up

You need Python 3.10 or newer and Chrome:

mkdir foodnow-selenium && cd foodnow-selenium
python -m venv .venv && source .venv/bin/activate
pip install selenium==4.52.0 pytest==9.2.0
mkdir -p site tests
printf '[pytest]\npythonpath = .\n' > pytest.ini

Step 2: The page under test

The FoodNow order page lets you add dishes, remove them, and see the total. Like many real pages, it gets the total from the server, so it shows "…" for a moment (800 ms here) before the amount appears. And there's a cookie banner over everything until you press OK. Save it as site/index.html:

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>FoodNow: your order</title>
<style>
  /* ...layout styles... */
  #cookies { position: fixed; inset: 0; background: rgba(0,0,0,.45); display: flex; align-items: flex-end; }
</style>
</head>
<body>
<header>FoodNow</header>
<main>
  <section>
    <h2>Menu</h2>
    <div class="item"><span>Paneer wrap · ₹180</span><button data-testid="add-wrap" data-name="Paneer wrap" data-price="180">Add</button></div>
    <div class="item"><span>Masala chai · ₹40</span><button data-testid="add-chai" data-name="Masala chai" data-price="40">Add</button></div>
    <div class="item"><span>Veg biryani · ₹260</span><button data-testid="add-biryani" data-name="Veg biryani" data-price="260">Add</button></div>
  </section>
  <section>
    <h2>Your order</h2>
    <div id="cart"></div>
    <p>Total: <span id="total">₹0</span></p>
    <button id="place">Place order</button>
    <p id="status" role="status"></p>
  </section>
</main>
<div id="cookies"><div><span>We use cookies to remember your basket.</span><button id="accept">OK</button></div></div>
<script>
  const cart = [];
  const DELIVERY = 40, FREE_FROM = 500;

  function render() {
    const el = document.getElementById("cart");
    el.innerHTML = "";
    cart.forEach((item, i) => {
      const row = document.createElement("div");
      row.className = "line";
      row.dataset.testid = "cart-line";
      row.innerHTML = `<span>${item.name}</span><button>Remove</button>`;
      row.querySelector("button").onclick = () => { cart.splice(i + 1, 1); render(); };
      el.appendChild(row);
    });
    // The total comes from the server (prices, offers, delivery), so it arrives a little later.
    document.getElementById("total").textContent = "…";
    setTimeout(() => {
      const food = cart.reduce((s, x) => s + x.price, 0);
      const total = food === 0 ? 0 : food + (food >= FREE_FROM ? 0 : DELIVERY);
      document.getElementById("total").textContent = `₹${total}`;
    }, 800);
  }

  document.querySelectorAll("[data-testid^=add-]").forEach(b =>
    b.onclick = () => { cart.push({ name: b.dataset.name, price: Number(b.dataset.price) }); render(); });
  document.getElementById("accept").onclick = () => document.getElementById("cookies").remove();
  document.getElementById("place").onclick = () =>
    document.getElementById("status").textContent = cart.length ? "Order placed" : "Your basket is empty";
</script>
</body>
</html>

(We left out the layout styles here to keep it short.) One bug is planted. See if you can spot it before the tests do.

Step 3: Fixtures for the browser and the page

tests/conftest.py serves the site folder on a free local port and opens Chrome for each test:

import functools
import http.server
import threading
from pathlib import Path

import pytest
from selenium import webdriver

SITE = Path(__file__).parents[1] / "site"


@pytest.fixture(scope="session")
def base_url():
    handler = functools.partial(http.server.SimpleHTTPRequestHandler, directory=SITE)
    server = http.server.ThreadingHTTPServer(("127.0.0.1", 0), handler)
    threading.Thread(target=server.serve_forever, daemon=True).start()
    yield f"http://127.0.0.1:{server.server_port}"
    server.shutdown()


@pytest.fixture
def driver():
    options = webdriver.ChromeOptions()
    options.add_argument("--headless=new")
    options.add_argument("--window-size=1280,800")
    driver = webdriver.Chrome(options=options)
    yield driver
    driver.quit()

webdriver.Chrome() is all it takes: Selenium Manager finds Chrome and the driver. --headless=new runs Chrome without a window, which is what you want in CI. Remove it to watch the test run. A fixed window size makes the layout, and so the test, the same on every machine.

Step 4: The first test, and the first error

Here's the test most people write first. Add a wrap, check the total. Save it as tests/test_first.py:

from selenium.webdriver.common.by import By


def test_add_a_wrap(driver, base_url):
    driver.get(base_url)
    driver.find_element(By.CSS_SELECTOR, "[data-testid=add-wrap]").click()
    total = driver.find_element(By.ID, "total").text
    assert total == "₹220"  # ₹180 wrap + ₹40 delivery

$ pytest tests/test_first.py
...
E       selenium.common.exceptions.ElementClickInterceptedException: Message: element click intercepted: Element <button data-testid="add-wrap" data-name="Paneer wrap" data-price="180">...</button> is not clickable at point (335, 163). Other element would receive the click: <div id="cookies">...</div>
E         (Session info: chrome=148.0.7778.215); For documentation on this error, please visit: https://www.selenium.dev/documentation/webdriver/troubleshooting/errors#elementclickinterceptedexception
...
============================== 1 failed in 2.12s ===============================

ElementClickInterceptedException. The button exists, but Selenium clicks like a person: at a point on the screen. At that point is the cookie banner, and the message even names it: Other element would receive the click: <div id="cookies">. Read the whole message. It usually tells you exactly what's in the way: a cookie banner, a chat widget, a sticky header, or a loading overlay.

Close the banner first, as a user would:

    driver.get(base_url)
    driver.find_element(By.ID, "accept").click()  # close the cookie banner first
    driver.find_element(By.CSS_SELECTOR, "[data-testid=add-wrap]").click()

Don't "fix" it with a JavaScript click (driver.execute_script("arguments[0].click()", button)). It clicks straight through the banner, which no user can do, so the test stops matching reality.

Step 5: The second error: reading too early

$ pytest tests/test_first.py
...
E       AssertionError: assert '…' == '₹220'
E
E         - ₹220
E         + …
============================== 1 failed in 1.00s ===============================

The click worked. Selenium then read the total immediately, while the page was still showing "…". The amount arrived 800 ms later, after the test had already failed.

The tempting fix is time.sleep(1). It passes today. It's also too slow on a fast machine and too short on a slow CI runner or a slow server day. A suite full of sleeps is both slow and flaky.

The right fix is an explicit wait: wait until a condition is true, up to a time limit, and continue the moment it is:

from selenium.webdriver.support.ui import WebDriverWait

WebDriverWait(driver, timeout=5).until(
    lambda d: d.find_element(By.ID, "total").text.startswith("₹")
)

This waits for "a real amount", not for "₹220". If the total is wrong, the wait still ends, and the assert afterwards shows the wrong value. If you wait for the exact expected text instead, a wrong total gives you a TimeoutException that doesn't say what the total was.

Selenium also has an implicit wait (driver.implicitly_wait(5)), which makes every find_element retry until the element appears. It wouldn't have helped here: the total element existed all along, it just had the wrong text. And mixing implicit and explicit waits can make timeouts unpredictable. Use explicit waits.

Step 6: A page object, and the real tests

As tests grow, the same locators and waits get copied into every test. A page object puts them in one class, so tests read like what a user does, and a change to the page is fixed in one place. Save this as tests/pages.py:

"""A page object: one place that knows how the order page is built."""
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait


class OrderPage:
    def __init__(self, driver, base_url):
        self.driver = driver
        self.base_url = base_url
        self.wait = WebDriverWait(driver, timeout=5)

    def open(self):
        self.driver.get(self.base_url)
        self.wait.until(EC.element_to_be_clickable((By.ID, "accept"))).click()
        self.wait.until(EC.invisibility_of_element_located((By.ID, "cookies")))
        return self

    def add(self, dish: str):
        self.driver.find_element(By.CSS_SELECTOR, f"[data-testid=add-{dish}]").click()
        return self

    def remove(self, name: str):
        for line in self.driver.find_elements(By.CSS_SELECTOR, "[data-testid=cart-line]"):
            if line.find_element(By.TAG_NAME, "span").text == name:
                line.find_element(By.TAG_NAME, "button").click()
                return self
        raise AssertionError(f"{name} is not in the basket")

    def items(self) -> list[str]:
        return [e.text for e in self.driver.find_elements(By.CSS_SELECTOR, "[data-testid=cart-line] span")]

    def total(self) -> str:
        # The total shows "…" while the price is being checked. Wait until it's a real amount.
        self.wait.until(lambda d: d.find_element(By.ID, "total").text.startswith("₹"))
        return self.driver.find_element(By.ID, "total").text

    def place_order(self) -> str:
        self.driver.find_element(By.ID, "place").click()
        self.wait.until(lambda d: d.find_element(By.ID, "status").text != "")
        return self.driver.find_element(By.ID, "status").text

The locators use data-testid attributes and ids, not long CSS paths or XPath like /html/body/main/section[2]/div[1]/button. Those break the moment someone adds a <div>.

The tests, in tests/test_order.py (delete test_first.py now):

import pytest

from tests.pages import OrderPage


@pytest.fixture
def order(driver, base_url):
    return OrderPage(driver, base_url).open()


def test_one_wrap_pays_delivery(order):
    order.add("wrap")
    assert order.total() == "₹220"  # ₹180 + ₹40 delivery


def test_free_delivery_from_500(order):
    order.add("biryani").add("biryani")
    assert order.total() == "₹520"  # ₹520 of food, delivery free


def test_remove_an_item(order):
    order.add("wrap").add("chai")
    order.remove("Paneer wrap")
    assert order.items() == ["Masala chai"]
    assert order.total() == "₹80"  # ₹40 + ₹40 delivery


def test_cannot_order_an_empty_basket(order):
    assert order.place_order() == "Your basket is empty"

Step 7: Take a screenshot when a test fails

A failing UI test without a screenshot sends you guessing. Add this hook to the end of tests/conftest.py. It saves a screenshot of the browser whenever a test that uses driver fails:

@pytest.hookimpl(wrapper=True)
def pytest_runtest_makereport(item, call):
    report = yield
    if report.when == "call" and report.failed and "driver" in item.fixturenames:
        path = Path("shots") / f"{item.name}.png"
        path.parent.mkdir(exist_ok=True)
        item.funcargs["driver"].save_screenshot(str(path))
        report.sections.append(("screenshot", str(path)))
    return report

Step 8: Run it

$ pytest tests -v
tests/test_order.py::test_one_wrap_pays_delivery PASSED                  [ 25%]
tests/test_order.py::test_free_delivery_from_500 PASSED                  [ 50%]
tests/test_order.py::test_remove_an_item FAILED                          [ 75%]
tests/test_order.py::test_cannot_order_an_empty_basket PASSED            [100%]

=================================== FAILURES ===================================
...
E       AssertionError: assert ['Paneer wrap'] == ['Masala chai']
E
E         At index 0 diff: 'Paneer wrap' != 'Masala chai'
E
E         Full diff: (-: missing in left side, +: extra in left side)
E           [
E         -     'Masala chai',
E         +     'Paneer wrap',
E           ]
---------------------------------- screenshot ----------------------------------
shots/test_remove_an_item.png
=========================== short test summary info ============================
FAILED tests/test_order.py::test_remove_an_item - AssertionError: assert ['Pa...
========================= 1 failed, 3 passed in 4.66s ==========================

We removed the paneer wrap, and the masala chai disappeared instead. The screenshot shows it:

(The total shows "…" because the screenshot was taken the moment the test failed, while the page was still working out the new total.)

The bug is in the Remove handler: cart.splice(i + 1, 1) removes the item after the one you clicked. It should be cart.splice(i, 1). With only one item in the basket, Remove does nothing at all. The three other tests passed, because none of them removes anything. That's why each user action needs at least one test of its own.

After the fix, we ran the suite five times in a row:

4 passed in 5.56s
4 passed in 7.70s
4 passed in 5.66s
4 passed in 5.63s
4 passed in 5.54s

No flaky failures. The waits are doing their job.

Selenium or Playwright?

We wrote two of the same tests with Playwright for Python (our first Playwright post explains the setup):

def test_remove_an_item(page):
    page.get_by_test_id("add-wrap").click()
    page.get_by_test_id("add-chai").click()
    page.get_by_test_id("cart-line").filter(has_text="Paneer wrap").get_by_role("button", name="Remove").click()
    expect(page.get_by_test_id("cart-line").locator("span")).to_have_text(["Masala chai"])
    expect(page.locator("#total")).to_have_text("₹80")

No explicit waits: Playwright waits for an element to be ready before clicking, and expect(...) retries until the text matches or 5 seconds pass. On the buggy page:

E       AssertionError: Locator expected to have text '['Masala chai']'
E       Actual value: ['Paneer wrap']
...
1 failed, 1 passed in 7.68s

On the fixed page, 2 passed in about 3 to 4 seconds. (Our first Playwright attempt also failed on the fixed page: to_have_text on the whole line compared "Masala chaiRemove", including the button's text. Every tool has its traps.)

Choose Selenium when your team already has a Selenium suite, needs real Safari or the widest set of languages, or runs on a device cloud built around WebDriver. Choose Playwright for a new suite where speed of writing and built-in waiting matter most. Both find the same bug, as long as someone writes the test that removes something.

Common mistakes

• time.sleep() everywhere. Slow when the page is fast, flaky when it's slow. Wait for a condition with WebDriverWait.
• Waiting for the expected value. A wrong value then gives a timeout instead of the actual value. Wait for "ready", then assert.
• JavaScript clicks to get past overlays. They click through things users can't. Close the overlay, as a user would.
• Absolute XPath and long CSS paths. Use ids, data-testid attributes, or text and roles users see.
• Locators copied into every test. Use page objects, so a change to the page is fixed in one place.
• No screenshot on failure. Add the hook from Step 7 on day one.
• Mixing implicit and explicit waits. It makes timeouts unpredictable. Pick explicit waits.

What's next

In Part 12 of Testing in Practice we write your first Cypress test: the same FoodNow order page, the Cypress way, and what's different about a tool that runs inside the browser.

ProvaLab's QA Infrastructure runs Selenium tests like these on browsers in the cloud, with a live view of the browser while the test runs. ProvaLab's pytest plugin sends each result to the ProvaLab test management system, with the screenshot attached and the test's history across runs. See it in the two-minute product tour.

This post tests a small Android app for FoodNow, our food delivery example, with Appium and pytest on an Android emulator. Two bugs are planted in it. While running the tests we found a third that we didn't plant, and two traps in the tooling that make tests lie. Every command and line of output below was run while writing this post.

What's different about testing a mobile app

This post covers the first four. The tools are the same for the rest.

How Appium works

Appium is a server that drives phone apps the way Selenium and WebDriver drive browsers. Your test sends commands over HTTP. Appium's UiAutomator2 driver installs your app and a small helper app on the phone, and the helper app taps, types and reads the screen.

The same test runs on an emulator or a real phone. Only the device changes.

Step 1: Set up

You need an Android emulator, an Appium server with the UiAutomator2 driver, and the Python client.

On Linux with KVM, the quickest way to get the first two is one Docker container that has both. We used this:

docker run -d --name foodnow-emu --device /dev/kvm \
  -p 4723:4723 -p 6080:6080 -v "$PWD":/apps:ro \
  -e EMULATOR_DEVICE="Samsung Galaxy S10" -e WEB_VNC=true -e APPIUM=true \
  budtmo/docker-android:emulator_14.0

It starts Android 14 with Appium on port 4723, and shows the phone's screen in your browser at http://localhost:6080. It needs a few minutes to boot the first time. The -v line makes the current folder visible inside the container as /apps, which is where Appium will look for the app.

On a Mac or Windows, use Android Studio's emulator instead, and install Appium yourself:

npm install -g appium
appium driver install uiautomator2
appium

Then the Python side:

pip install Appium-Python-Client==6.1.5 pytest==9.2.0

We used Appium 3.9.0 with the UiAutomator2 driver 8.7.3 (both inside the container), an Android 14 emulator (API 34), Python 3.11, pytest 9.2.0 and Appium-Python-Client 6.1.5, on Linux.

Step 2: The app

The app is one checkout screen: two paneer wraps for 500, a box for the delivery tip, Add tip and Remove tip buttons, the total, and Pay. Every control has an id in the layout, such as android:id="@+id/total", which Appium sees as the resource id com.example.foodnow:id/total. Ids are the most reliable way to find elements, like data-testid on the web.

Here is the code behind the screen, with the two planted bugs in it:

public class CheckoutActivity extends Activity {
    private static final int SUBTOTAL = 500;
    private int tip = 0;
    private TextView total;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.checkout);
        EditText tipField = findViewById(R.id.tip);
        TextView status = findViewById(R.id.status);
        total = findViewById(R.id.total);
        showTotal();

        findViewById(R.id.add_tip).setOnClickListener(v -> {
            String text = tipField.getText().toString();
            int value = text.isEmpty() ? 0 : Integer.parseInt(text);
            if (value > 200) {
                status.setText("Tip must be 0 to 200");
                return;
            }
            tip = value;
            status.setText("");
            showTotal();
        });
        findViewById(R.id.remove_tip).setOnClickListener(v -> {
            tip = 0;
            tipField.setText("");
            showTotal();
        });
        findViewById(R.id.pay).setOnClickListener(v ->
                status.setText("Order placed: " + (SUBTOTAL + tip)));
    }

    private void showTotal() {
        total.setText("Total: " + (SUBTOTAL + tip));
    }
}

We built it into foodnow.apk with the Android SDK's own tools. In a real project, Android Studio or ./gradlew assembleDebug builds the APK for you.

Step 3: A fixture that starts the app, and collects evidence

tests/conftest.py opens an Appium session for each test, with the app freshly installed and its data cleared. It also does two things most examples leave out. Both turned out to matter.

import os
import pathlib

import pytest
from appium import webdriver
from appium.options.android import UiAutomator2Options

APP = os.environ.get("APP", str(pathlib.Path(__file__).parent.parent / "foodnow.apk"))
APPIUM_URL = os.environ.get("APPIUM_URL", "http://127.0.0.1:4723")


@pytest.hookimpl(wrapper=True)
def pytest_runtest_makereport(item, call):
    """Remember whether the test failed, so the fixture can collect evidence."""
    report = yield
    setattr(item, f"rep_{report.when}", report)
    return report


@pytest.fixture
def driver(request):
    options = UiAutomator2Options()
    options.app = APP
    options.app_package = "com.example.foodnow"
    options.app_activity = ".CheckoutActivity"
    options.no_reset = False  # fresh app data for every test
    options.set_capability("appium:enforceAppInstall", True)  # always install the APK you just built
    driver = webdriver.Remote(APPIUM_URL, options=options)
    driver.implicitly_wait(5)
    driver.get_log("logcat")  # skip old log lines: this returns, and clears, what's there so far
    yield driver
    if getattr(request.node, "rep_call", None) and request.node.rep_call.failed:
        # A crash on a phone shows up only as "element not found". Print the real cause.
        crash = [e["message"] for e in driver.get_log("logcat") if " E AndroidRuntime" in e["message"]]
        print("\n".join(crash[:4]) or "no crash in logcat")
        driver.save_screenshot(f"{request.node.name}.png")
    driver.quit()

When a test fails, it prints the crash lines from the phone's log (logcat) and saves a screenshot. Step 5 shows why.

Step 4: The tests

The everyday checks, in tests/test_checkout.py. The last one is a boundary test: a tip far too long to be a real number.

from appium.webdriver.common.appiumby import AppiumBy

PKG = "com.example.foodnow:id/"


def el(driver, name):
    return driver.find_element(AppiumBy.ID, PKG + name)


def add_tip(driver, amount):
    el(driver, "tip").send_keys(str(amount))
    el(driver, "add_tip").click()


def test_tip_is_added_to_the_total(driver):
    add_tip(driver, 50)
    assert el(driver, "total").text == "Total: 550"


def test_remove_tip(driver):
    add_tip(driver, 50)
    el(driver, "remove_tip").click()
    assert el(driver, "total").text == "Total: 500"


def test_tip_over_200_is_refused(driver):
    add_tip(driver, 250)
    assert el(driver, "status").text == "Tip must be 0 to 200"
    assert el(driver, "total").text == "Total: 500"


def test_a_huge_tip_is_refused_not_a_crash(driver):
    add_tip(driver, 99999999999)
    assert el(driver, "status").text == "Tip must be 0 to 200"

And one test that only makes sense on a phone, in tests/test_mobile_only.py: add a tip, turn the phone sideways, then pay.

from appium.webdriver.common.appiumby import AppiumBy
from selenium.common.exceptions import StaleElementReferenceException
from selenium.webdriver.support.ui import WebDriverWait

from test_checkout import PKG, add_tip, el


def text_of(driver, name):
    """Read a text, waiting out the moment Android rebuilds the screen."""
    return WebDriverWait(driver, 5, ignored_exceptions=[StaleElementReferenceException]).until(
        lambda d: el(d, name).text
    )


def scroll_to(driver, name):
    """Scroll until the element is on screen, like a person would. Fails if nothing scrolls.

    The dead zone keeps the swipe away from the screen edges. Without it, the swipe starts
    at the bottom edge, and Android's gesture navigation takes it as "go to recent apps".
    """
    driver.find_element(
        AppiumBy.ANDROID_UIAUTOMATOR,
        "new UiScrollable(new UiSelector().scrollable(true)).setSwipeDeadZonePercentage(0.25)"
        f'.scrollIntoView(new UiSelector().resourceId("{PKG}{name}"))',
    )
    return el(driver, name)


def test_tip_survives_rotating_the_phone(driver):
    add_tip(driver, 50)

    driver.orientation = "LANDSCAPE"

    assert text_of(driver, "tip") == "50"
    assert text_of(driver, "total") == "Total: 550"
    scroll_to(driver, "pay").click()
    assert text_of(driver, "status") == "Order placed: 550"

text_of is there because rotating rebuilds the screen. Our first version read the text straight away, and failed with StaleElementReferenceException: The element 'By.id: com.example.foodnow:id/tip' does not exist in DOM anymore: Appium had found the old screen just before Android threw it away. The two notes in scroll_to come from Step 7.

Step 5: The run

The Appium server runs inside the container, so we tell the tests where the app is from the container's point of view:

$ APP=/apps/foodnow.apk python -m pytest tests --tb=short
============================= test session starts ==============================
platform linux -- Python 3.11.17, pytest-9.2.0, pluggy-1.7.0
rootdir: ~/foodnow-mobile
collected 5 items

tests/test_checkout.py ...F                                              [ 80%]
tests/test_mobile_only.py F                                              [100%]

=================================== FAILURES ===================================
____________________ test_a_huge_tip_is_refused_not_a_crash ____________________
tests/test_checkout.py:34: in test_a_huge_tip_is_refused_not_a_crash
    assert el(driver, "status").text == "Tip must be 0 to 200"
           ^^^^^^^^^^^^^^^^^^^^
tests/test_checkout.py:7: in el
    return driver.find_element(AppiumBy.ID, PKG + name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...
E   selenium.common.exceptions.NoSuchElementException: Message: An element could not be located on the page using the given search parameters.
...
--------------------------- Captured stdout teardown ---------------------------
11-11 07:31:31.853 24324 24324 E AndroidRuntime: FATAL EXCEPTION: main
11-11 07:31:31.853 24324 24324 E AndroidRuntime: Process: com.example.foodnow, PID: 24324
11-11 07:31:31.853 24324 24324 E AndroidRuntime: java.lang.NumberFormatException: For input string: "99999999999"
11-11 07:31:31.853 24324 24324 E AndroidRuntime: 	at java.lang.Integer.parseInt(Integer.java:797)
_____________________ test_tip_survives_rotating_the_phone _____________________
tests/test_mobile_only.py:35: in test_tip_survives_rotating_the_phone
    assert text_of(driver, "total") == "Total: 550"
E   AssertionError: assert 'Total: 500' == 'Total: 550'
E
E     - Total: 550
E     ?        -
E     + Total: 500
E     ?          +
--------------------------- Captured stdout teardown ---------------------------
no crash in logcat
=========================== short test summary info ============================
FAILED tests/test_checkout.py::test_a_huge_tip_is_refused_not_a_crash - selen...
FAILED tests/test_mobile_only.py::test_tip_survives_rotating_the_phone - Asse...
========================= 2 failed, 3 passed in 45.07s =========================

(We cut the lines from inside Selenium and the Appium server, marked ....)

Bug 1: the app crashed

The test's own error says only "element not found". On a phone, that's what a crash looks like from the outside: the app is gone, so every element is gone. Without the crash lines from logcat, you'd be looking for a wrong locator. The real cause is in the teardown output: NumberFormatException: For input string: "99999999999".

The tip box is set to inputType="number", which shows a number keyboard but doesn't limit the length. Integer.parseInt can't hold a number larger than 2,147,483,647, so it throws, nothing catches it, and Android closes the app. The screenshot the fixture saved:

The three everyday tests passed. Only the boundary test, a value at the extreme, found this. That's the same lesson as boundary value analysis anywhere, with a harsher result on a phone.

Bug 2: the tip disappeared when the phone turned

After rotating, the tip box still says 50, but the total says 500. If the customer taps Pay now, they pay 500 and the courier's tip is gone, while the screen still shows the 50 they typed.

When the phone rotates, Android destroys the screen (the Activity) and creates it again. Android saves the text of an EditText that has an id for you, so the tip box comes back with 50 in it. The tip variable in our code isn't saved, so it starts again at 0, and showTotal() shows 500. The two halves of the screen now disagree.

You can't find this by testing in one orientation, and a tester tapping through on a desk rarely turns the phone at exactly the wrong moment.

Step 6: Fix both

Save the tip when Android is about to destroy the screen, bring it back when it rebuilds it, and treat a number too long to read as too big:

     protected void onCreate(Bundle savedInstanceState) {
         super.onCreate(savedInstanceState);
         setContentView(R.layout.checkout);
+        if (savedInstanceState != null) {
+            tip = savedInstanceState.getInt("tip");  // the screen was rebuilt: bring the tip back
+        }
         EditText tipField = findViewById(R.id.tip);
@@
         findViewById(R.id.add_tip).setOnClickListener(v -> {
             String text = tipField.getText().toString();
-            int value = text.isEmpty() ? 0 : Integer.parseInt(text);
+            int value;
+            try {
+                value = text.isEmpty() ? 0 : Integer.parseInt(text);
+            } catch (NumberFormatException e) {
+                value = Integer.MAX_VALUE;  // too long to be a number: treat it as too big
+            }
             if (value > 200) {
@@
+    @Override
+    protected void onSaveInstanceState(Bundle outState) {
+        super.onSaveInstanceState(outState);
+        outState.putInt("tip", tip);
+    }

Trap 1: the fix that never reached the phone

We rebuilt the app and ran the tests again. Same two failures. The Appium server's log had the reason:

[ADB] The version name of the installed 'com.example.foodnow' is greater or equal to the application version name ('1.0.0' >= '1.0.0')
[ADB] There is no need to install/upgrade '/apps/foodnow.apk'

Appium saw an app with the same version already on the phone and skipped installing the new build. We were testing the old code. This is why the fixture in Step 3 sets enforceAppInstall. Without it, bump the version number on every build, or uninstall the app before the run.

With the new build actually installed:

tests/test_checkout.py ....                                              [ 80%]
tests/test_mobile_only.py F                                              [100%]
...
tests/test_mobile_only.py:36: in test_tip_survives_rotating_the_phone
    scroll_to(driver, "pay").click()
tests/test_mobile_only.py:21: in scroll_to
...
E   selenium.common.exceptions.NoSuchElementException: Message: An element could not be located on the page using the given search parameters.
========================= 1 failed, 4 passed in 50.07s =========================

The crash is fixed, and the total now survives rotation. But the test can't get to Pay.

Step 7: Bug 3, the Pay button you can't reach

We didn't plant this one. In landscape, the screen is 1,440 pixels tall, and the checkout is taller than that. Pay is below the bottom edge, and the layout is a plain LinearLayout, which doesn't scroll. In landscape, the customer can't pay at all. In portrait it fits, so the three everyday tests never noticed.

The fix is to wrap the layout in a ScrollView:

<ScrollView xmlns:android="http://schemas.android.com/apk/res/android"
    android:layout_width="match_parent" android:layout_height="match_parent">
<LinearLayout
    android:layout_width="match_parent" android:layout_height="wrap_content"
    android:orientation="vertical" android:padding="24dp">
    ...
</LinearLayout>
</ScrollView>

Trap 2: the swipe that left the app

With the ScrollView in, our first version of scroll_to, without the dead zone, still failed. The screenshot showed why: the whole app had shrunk into a card in Android's recent-apps view.

The emulator uses gesture navigation: a swipe up from the bottom edge of the screen means "go home" or "show recent apps". The ScrollView reaches the bottom edge, so the scroll swipe started right on it, and Android took it as a system gesture, not a scroll. setSwipeDeadZonePercentage(0.25) makes Appium start and end its swipes a quarter of the way in from the edges. Most new Android phones use gesture navigation by default, so this catches you on real devices too.

The full run, after all three fixes:

$ APP=/apps/foodnow.apk python -m pytest tests
============================= test session starts ==============================
platform linux -- Python 3.11.17, pytest-9.2.0, pluggy-1.7.0
rootdir: ~/foodnow-mobile
collected 5 items

tests/test_checkout.py ....                                              [ 80%]
tests/test_mobile_only.py .                                              [100%]

============================== 5 passed in 53.88s ==============================

We ran it four more times on the final build: 5 passed every time, in 51 to 60 seconds. And to make sure the scroll test catches what it should, we put the old layout without the ScrollView back: the test failed again at scroll_to.

Emulators or real phones?

Emulators are free, quick to reset and easy to run in CI. They're the right place for most functional tests, like these. Real phones find what emulators can't: a maker's changes to Android, real performance and battery use, the camera, fingerprint sensor and mobile network, and the odd screen shapes. Most teams run the bulk on emulators and a smaller set on a few real phones that their users actually have.

Common mistakes

• Testing only in portrait. Rotate in at least one test for every screen that takes input. Rotation is the cheapest way to find lost state.
• Reading "element not found" at face value. On a phone it often means the app crashed. Collect logcat and a screenshot on every failure, as the fixture in Step 3 does.
• Testing an old build without knowing it. Appium skips installing an app with the same version. Use enforceAppInstall, or change the version on every build.
• Fixed sleeps after rotating or navigating. Wait for the condition instead, ignoring stale elements, as text_of does.
• Swipes at the screen edges. Gesture navigation takes them. Keep swipes away from the edges.
• Locating by text or position. Text changes with language, and position changes with screen size. Use resource ids, or accessibility ids for things like icon buttons.

What's next

In Part 7 of Testing in Practice we look at contract testing: how to check that an app and the API it calls still agree, without running both together, using Pact.

ProvaLab's QA Infrastructure runs Appium tests like these on Android emulators and real phones, with a live view of the device while the test runs, and sends each result to the ProvaLab test management system. See it in the two-minute product tour.

Test doubles make tests fast and reliable. They can also make tests lie. A double only knows what you told it, and you told it the same thing you believed when you wrote the code. If that belief is wrong, the code and the double are wrong together, and the test passes.

This post shows the five kinds of double with one example. Then it runs a real experiment: the same checkout code, tested with four kinds of double, with two bugs planted in it. Every command and line of output below was run while writing this post.

The five kinds of test double

The names come from Gerard Meszaros's book xUnit Test Patterns (2007). People often call all of them "mocks", but they do different jobs:

In Python, unittest.mock.MagicMock can act as a dummy, a stub, a spy or a mock depending on how you use it. A fake is usually a small class you write yourself.

Step 1: The code under test

You need Python 3.10 or newer and pytest:

mkdir foodnow-pay && cd foodnow-pay
python -m venv .venv && source .venv/bin/activate
pip install pytest==9.2.0

We used Python 3.11 and pytest 9.2.0 on Linux. unittest.mock is part of Python, so there's nothing else to install.

gateway.py is the client for the payment provider. It sends one HTTP request and returns a Payment with a status of "succeeded" or "declined". The amount is in paise, the smallest unit of the rupee, and every charge needs an idempotency key, so the provider can tell a retry from a new charge.

import json
import os
import urllib.request
from dataclasses import dataclass

PAYMENTS_URL = os.environ.get("PAYMENTS_URL", "https://api.payments.example")


@dataclass
class Payment:
    id: str
    status: str  # "succeeded" or "declined"


class PaymentGateway:
    """Client for the payment provider's HTTP API."""

    def charge(self, amount_paise: int, idempotency_key: str) -> Payment:
        request = urllib.request.Request(
            f"{PAYMENTS_URL}/v1/charges",
            data=json.dumps({"amount": amount_paise, "currency": "INR"}).encode(),
            headers={"Content-Type": "application/json", "Idempotency-Key": idempotency_key},
            method="POST",
        )
        with urllib.request.urlopen(request, timeout=5) as response:
            body = json.load(response)
        return Payment(id=body["id"], status=body["status"])

(.example is a domain reserved for examples, so this URL never reaches anyone.)

checkout.py charges the customer and marks the order paid or failed. The developer wrote it from memory of an older version of the client:

from gateway import PaymentGateway


def pay_for_order(order, gateway=None):
    gateway = gateway or PaymentGateway()
    payment = gateway.charge(amount=order["total"])
    if payment.ok:
        order["status"] = "paid"
    else:
        order["status"] = "payment_failed"
    return order

It has two bugs:

• Bug A, wrong arguments. The client takes amount_paise and idempotency_key. This passes amount=, in rupees, and no key.
• Bug B, a field that isn't there. Payment has a status, not an ok.

In production, the first bug makes every checkout crash before charging anyone. Fix only that one, and the second makes every checkout crash after the customer was charged, with the order still unpaid.

Step 2: MagicMock: both bugs, all green

The most common way to test this is a MagicMock in place of the gateway:

from unittest.mock import MagicMock

from checkout import pay_for_order


def test_successful_payment_marks_the_order_paid():
    gateway = MagicMock()

    order = pay_for_order({"id": 7, "total": 550}, gateway)

    assert order["status"] == "paid"
    gateway.charge.assert_called_once()


def test_declined_payment_marks_the_order_failed():
    gateway = MagicMock()
    gateway.charge.return_value.ok = False

    order = pay_for_order({"id": 8, "total": 550}, gateway)

    assert order["status"] == "payment_failed"

$ python -m pytest -q tests/test_with_magicmock.py
..                                                                       [100%]
2 passed in 0.02s

Both pass, with both bugs in the code. Two reasons:

1. A MagicMock accepts any call. charge(amount=550) is fine, and so is charge(banana=True). It has no idea what the real charge looks like.
2. Any field you ask a MagicMock for is another MagicMock, and a MagicMock counts as true. payment.ok doesn't exist on a real Payment, but on the mock it's there, and it's truthy, so the order is "paid".

Look at the second test again: gateway.charge.return_value.ok = False. The person who wrote the code believed a payment has an ok field, and they wrote that same belief into the test. The mock agrees with the code because the same person wrote both.

Step 3: An autospec mock catches the wrong arguments

create_autospec builds a mock from the real class. It only allows the methods the class has, called the way the class defines them:

from unittest.mock import create_autospec

from checkout import pay_for_order
from gateway import PaymentGateway


def test_successful_payment_marks_the_order_paid():
    gateway = create_autospec(PaymentGateway, instance=True)

    order = pay_for_order({"id": 7, "total": 550}, gateway)

    assert order["status"] == "paid"

tests/test_with_autospec.py F                                            [100%]

=================================== FAILURES ===================================
_________________ test_successful_payment_marks_the_order_paid _________________
tests/test_with_autospec.py:10: in test_successful_payment_marks_the_order_paid
    order = pay_for_order({"id": 7, "total": 550}, gateway)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
checkout.py:6: in pay_for_order
    payment = gateway.charge(amount=order["total"])
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.../lib/python3.11/unittest/mock.py:1122: in __call__
    self._mock_check_sig(*args, **kwargs)
.../lib/python3.11/unittest/mock.py:131: in checksig
    sig.bind(*args, **kwargs)
.../lib/python3.11/inspect.py:3195: in bind
    return self._bind(args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
.../lib/python3.11/inspect.py:3110: in _bind
    raise TypeError(msg) from None
E   TypeError: missing a required argument: 'amount_paise'

(We shortened the path to Python's own files to .../lib/python3.11.)

Bug A is caught. The developer fixes the call:

    payment = gateway.charge(
        amount_paise=order["total"] * 100, idempotency_key=f"order-{order['id']}"
    )

Now the autospec test passes, and so do the MagicMock tests:

$ python -m pytest -q tests
...                                                                      [100%]
3 passed in 0.04s

Bug B is still there. An autospec mock checks how you call a method, but what the method returns is still a plain MagicMock, so payment.ok is still a truthy mock.

Step 4: A stub with a real answer catches the missing field

The fix for the test is to make the double answer with a real Payment, the same type of object the real client returns. Now it's a stub:

from unittest.mock import create_autospec

from checkout import pay_for_order
from gateway import Payment, PaymentGateway


def stub_gateway(status):
    """Checks the call matches the real signature, and answers with a real Payment."""
    gateway = create_autospec(PaymentGateway, instance=True)
    gateway.charge.return_value = Payment(id="ch_1", status=status)
    return gateway


def test_successful_payment_marks_the_order_paid():
    order = pay_for_order({"id": 7, "total": 550}, stub_gateway("succeeded"))
    assert order["status"] == "paid"


def test_declined_payment_marks_the_order_failed():
    order = pay_for_order({"id": 8, "total": 550}, stub_gateway("declined"))
    assert order["status"] == "payment_failed"

Both tests fail with AttributeError: 'Payment' object has no attribute 'ok': exactly what would happen in production.

Step 5: A fake, for the behaviour you can't stub

Some things a stub can't test, because they depend on what happened before. The provider uses the idempotency key so that a double click, or a retry after a timeout, doesn't charge the customer twice. To test that your code sends the same key for the same order, you need a double that remembers. That's a fake:

from gateway import Payment


class FakeGateway:
    """Works like the provider's sandbox, in memory: no network, same answers."""

    def __init__(self, decline=False):
        self.decline = decline
        self.charges = {}  # idempotency key -> (amount, Payment)

    def charge(self, amount_paise: int, idempotency_key: str) -> Payment:
        if not isinstance(amount_paise, int) or amount_paise <= 0:
            raise ValueError(f"amount must be a positive whole number of paise, got {amount_paise!r}")
        if idempotency_key in self.charges:  # a retry: same answer, no second charge
            return self.charges[idempotency_key][1]
        status = "declined" if self.decline else "succeeded"
        payment = Payment(id=f"ch_{len(self.charges) + 1}", status=status)
        self.charges[idempotency_key] = (amount_paise, payment)
        return payment

The tests read like a description of the business rules. The first also uses the fake as a spy: it checks the amount that was charged.

from checkout import pay_for_order
from fakes import FakeGateway


def test_successful_payment_marks_the_order_paid():
    gateway = FakeGateway()

    order = pay_for_order({"id": 7, "total": 550}, gateway)

    assert order["status"] == "paid"
    assert [amount for amount, _ in gateway.charges.values()] == [55000]


def test_declined_payment_marks_the_order_failed():
    gateway = FakeGateway(decline=True)

    order = pay_for_order({"id": 8, "total": 550}, gateway)

    assert order["status"] == "payment_failed"


def test_paying_twice_charges_the_customer_once():
    gateway = FakeGateway()

    pay_for_order({"id": 9, "total": 550}, gateway)
    pay_for_order({"id": 9, "total": 550}, gateway)  # double click, or a retry

    assert len(gateway.charges) == 1

With Bug A fixed and Bug B still in, all three fail at the same line:

tests/test_with_fake.py FFF                                              [ 66%]
...
checkout.py:9: in pay_for_order
    if payment.ok:
       ^^^^^^^^^^
E   AttributeError: 'Payment' object has no attribute 'ok'

Step 6: Fix the code, and watch the mocks fail

The fix for Bug B is one line:

    if payment.status == "succeeded":

The stub and fake tests now pass. And here is the part most people don't expect. We ran the old MagicMock and autospec tests against the correct code:

tests/test_with_autospec.py F                                            [ 16%]
tests/test_with_fake.py ...                                              [ 66%]
tests/test_with_magicmock.py F.                                          [100%]

=================================== FAILURES ===================================
_________________ test_successful_payment_marks_the_order_paid _________________
tests/test_with_autospec.py:12: in test_successful_payment_marks_the_order_paid
    assert order["status"] == "paid"
E   AssertionError: assert 'payment_failed' == 'paid'
E
E     - paid
E     + payment_failed
_________________ test_successful_payment_marks_the_order_paid _________________
tests/test_with_magicmock.py:11: in test_successful_payment_marks_the_order_paid
    assert order["status"] == "paid"
E   AssertionError: assert 'payment_failed' == 'paid'
E
E     - paid
E     + payment_failed
=========================== short test summary info ============================
FAILED tests/test_with_autospec.py::test_successful_payment_marks_the_order_paid
FAILED tests/test_with_magicmock.py::test_successful_payment_marks_the_order_paid
========================= 2 failed, 4 passed in 0.04s ==========================

The mock tests passed with the bugs and failed with the fix. A MagicMock's status is a MagicMock, not "succeeded", so the correct code marks the order as failed. These tests weren't checking the payment logic. They were checking that the code matched the mock.

Step 7: The patch that missed

There's one more common trap. This test doesn't pass a gateway in. It uses patch to replace the PaymentGateway class while the test runs:

from unittest.mock import patch

from checkout import pay_for_order
from gateway import Payment


@patch("gateway.PaymentGateway")
def test_paid_with_patched_gateway(gateway_class):
    gateway_class.return_value.charge.return_value = Payment(id="ch_1", status="succeeded")

    order = pay_for_order({"id": 7, "total": 550})  # no gateway passed: checkout makes its own

    assert order["status"] == "paid"

gateway.py:25: in charge
    with urllib.request.urlopen(request, timeout=5) as response:
...
E   urllib.error.URLError: <urlopen error [Errno -2] Name or service not known>
=========================== short test summary info ============================
FAILED tests/test_with_patch.py::test_paid_with_patched_gateway - urllib.erro...

(We cut the part of the traceback inside Python's network code.)

The test tried to make a real network request. checkout.py did from gateway import PaymentGateway when it was imported, so it has its own name for the class. Patching gateway.PaymentGateway changes the name in gateway, not the one checkout uses. Here the URL goes nowhere, so the test failed. With a real provider URL in your test settings, it would have sent a real charge to the provider.

Patch where the name is used, not where it's defined:

@patch("checkout.PaymentGateway")

$ python -m pytest tests/test_with_stub.py tests/test_with_fake.py tests/test_with_patch.py
============================= test session starts ==============================
platform linux -- Python 3.11.17, pytest-9.2.0, pluggy-1.7.0
rootdir: ~/foodnow-pay
collected 6 items

tests/test_with_stub.py ..                                               [ 33%]
tests/test_with_fake.py ...                                              [ 83%]
tests/test_with_patch.py .                                               [100%]

============================== 6 passed in 0.03s ===============================

Passing the gateway in as an argument, as the other tests do, avoids the trap completely. That's one reason people design code so its dependencies can be passed in.

Which double should you use?

• Answers only (stub): most of the time. Use create_autospec so calls are checked against the real signature, and return real objects, not mocks.
• Behaviour across calls (fake): retries, idempotency, "add to cart, then check out". Write it once and share it across tests.
• "Was it called?" (mock or spy): only when the call is the result, such as "an email was sent" or "the card was charged once". Prefer checking the outcome when you can.
• Nothing at all: your own code, simple data classes, and pure functions. Don't replace what you're testing.

How to keep doubles honest

A double is a copy of your beliefs about the real thing. Keep it close to the real thing:

1. Return real objects. A Payment, not a MagicMock, so missing fields fail like they would in production.
2. Use autospec. create_autospec, or patch(..., autospec=True), so a changed signature breaks the test, not production.
3. Test the fake against the real thing. Run a small set of the same tests against both the fake and the provider's sandbox, for example nightly. If the sandbox starts answering differently, the fake is out of date. This is the idea behind contract testing.
4. Keep some integration tests. A test suite where everything external is replaced proves the parts work as you imagine. A few tests against the real database or sandbox prove they work as they are. Part 4 found three bugs that no mock could have.

Common mistakes

• MagicMock everywhere. It accepts every call and every field, so it can't fail for the reasons production does. Step 2 passed with two bugs.
• Mocks that return mocks. gateway.charge.return_value.ok = False writes your assumption into the test. Return the real type.
• Checking calls instead of results. assert_called_once_with(...) for every step ties the test to how the code works, not what it achieves. Refactor the code and every test breaks, even when nothing is wrong.
• Patching in the wrong place. Patch where the name is looked up, as Step 7 shows.
• Faking your own code. If the test replaces the thing it's meant to test, it tests the double.

What's next

In Part 6 of Testing in Practice we test a mobile app: an Android emulator, Appium and pytest, on the FoodNow app.

ProvaLab's pytest plugin sends every result from tests like these to the ProvaLab test management system, with the history of each test across runs. So when the tests that were green with the bug go red with the fix, as in Step 6, you see that change in the test's history next to the code change that caused it. See it in the two-minute product tour.

Most test estimates are wrong in the same direction: too low. Not because testers are bad at it, but because of how the numbers are put together. This post estimates the testing for a new FoodNow feature step by step, and uses a short Python script to show where the usual method goes wrong and what to say instead.

The task estimates below are an example of what a test team might give for this feature. They're judgement, as all estimates are. Every number calculated from them, including the 100,000-run simulation, comes from running the scripts in this post.

Why test estimates go wrong

• Adding up the "most likely" times. Each task's most likely time is more likely than any other single value, but tasks overrun more than they underrun. Add nine of them up, and the total is very unlikely. Step 3 measures how unlikely.
• Forgetting the loop. Testing isn't one pass. You find bugs, they're fixed, you retest, and you run the regression tests again. Teams estimate the first pass and forget the rest.
• Testing waits on other work. A late build, a broken test environment or missing test data stops testing, and the clock keeps running.
• Turning hours into days too simply. Nobody tests for eight hours a day. Meetings, questions, and context switching take their share.

The techniques, briefly

This post uses the first three together.

Step 1: Break the work down

The feature is group ordering: an office manager starts a group order, colleagues add their own dishes from a link, and the manager pays once. Here are the testing tasks, each with three estimates in hours:

• O (optimistic): if everything goes well.
• M (most likely): what usually happens.
• P (pessimistic): if things go badly, but not a disaster.

Save them as tasks.csv:

task,optimistic,likely,pessimistic
Read the stories and ask questions,3,4,8
Write test cases (about 40),10,14,24
Set up test data: offices and 30 test users,3,5,12
Automate the 10 most important cases,12,16,30
First run of all cases,8,10,16
Report bugs and talk them through,3,5,12
Retest fixes and run regression,6,10,28
Exploratory testing (3 sessions),5,6,9
Test summary report,2,3,5

Look at "Retest fixes and run regression": 6 hours if the build is clean, 28 if it isn't. That one line is where most test estimates go wrong, and here it has its own row with an honest range.

Break tasks down until each one is small enough to picture doing, roughly a day or less. "Test group ordering: 70 hours" can't be checked. "Write about 40 test cases: 14 hours" can: that's about 20 minutes a case.

Step 2: Three-point estimates

The PERT formula turns three points into one expected value, with the most likely time counting four times:

expected = (O + 4M + P) / 6

Save this as estimate.py:

"""Three-point (PERT) estimate for a test effort, plus a simulation of the total."""
import csv
import random
import statistics
import sys

rows = list(csv.DictReader(open(sys.argv[1], encoding="utf-8")))
tasks = [(r["task"], float(r["optimistic"]), float(r["likely"]), float(r["pessimistic"])) for r in rows]

print(f"{'task':<42}{'O':>5}{'M':>5}{'P':>5}{'PERT':>7}")
for name, o, m, p in tasks:
    print(f"{name:<42}{o:>5.0f}{m:>5.0f}{p:>5.0f}{(o + 4 * m + p) / 6:>7.1f}")

sum_likely = sum(m for _, _, m, _ in tasks)
sum_pert = sum((o + 4 * m + p) / 6 for _, o, m, p in tasks)
print(f"\nsum of 'most likely':      {sum_likely:6.1f} hours")
print(f"sum of PERT expected:      {sum_pert:6.1f} hours")


def pert_sample(o: float, m: float, p: float) -> float:
    """One draw from the PERT (beta) distribution with these three points."""
    if p == o:
        return m
    a = 1 + 4 * (m - o) / (p - o)
    b = 1 + 4 * (p - m) / (p - o)
    return o + random.betavariate(a, b) * (p - o)


random.seed(13)
RUNS = 100_000
totals = sorted(sum(pert_sample(o, m, p) for _, o, m, p in tasks) for _ in range(RUNS))


def pct(q: float) -> float:
    return totals[int(q * RUNS) - 1]


print(f"\nsimulated {RUNS:,} projects:")
for q in (0.5, 0.8, 0.95):
    print(f"  {int(q * 100)}% chance of finishing within {pct(q):6.1f} hours")
within = sum(t <= sum_likely for t in totals) / RUNS
print(f"  chance of finishing within the sum of 'most likely' ({sum_likely:.0f} h): {within:.1%}")
print(f"  mean {statistics.mean(totals):.1f} h, standard deviation {statistics.stdev(totals):.1f} h")
if len(sys.argv) > 2:
    deadline = float(sys.argv[2])
    print(f"  chance of finishing within {deadline:.0f} hours: {sum(t <= deadline for t in totals) / RUNS:.1%}")

It needs only Python, no packages. The first part:

$ python estimate.py tasks.csv
task                                          O    M    P   PERT
Read the stories and ask questions            3    4    8    4.5
Write test cases (about 40)                  10   14   24   15.0
Set up test data: offices and 30 test users    3    5   12    5.8
Automate the 10 most important cases         12   16   30   17.7
First run of all cases                        8   10   16   10.7
Report bugs and talk them through             3    5   12    5.8
Retest fixes and run regression               6   10   28   12.3
Exploratory testing (3 sessions)              5    6    9    6.3
Test summary report                           2    3    5    3.2

sum of 'most likely':        73.0 hours
sum of PERT expected:        81.3 hours

Every PERT value is higher than the most likely value, because every task can overrun by more than it can underrun. Retesting can take 18 hours longer than likely, but only 4 hours less. Added up, the gap is 8.3 hours, more than a full working day.

Step 3: Simulate it 100,000 times

A single number hides the question that matters: how likely is it that we finish within X? To answer it, the script plays out the project 100,000 times. In each run, every task takes a random time drawn from its own three-point range, with times near the most likely value more common. Then it looks at how the totals spread out:

simulated 100,000 projects:
  50% chance of finishing within   81.0 hours
  80% chance of finishing within   86.6 hours
  95% chance of finishing within   92.1 hours
  chance of finishing within the sum of 'most likely' (73 h): 8.7%
  mean 81.3 h, standard deviation 6.3 h

The usual estimate, the sum of the most likely times, is 73 hours. In the simulation, the project finished within 73 hours 8.7% of the time. That's roughly one feature in eleven. The number most teams give is a number they'll almost always miss, even when every single task estimate was reasonable.

The number to give is the 80% point: 86.6 hours. It's not padding. It's the same estimates, added up correctly.

Step 4: When tasks slip together

The simulation above assumes each task goes well or badly on its own. In reality, many causes hit several tasks at once. A late, buggy build makes the first run slower, the bug reports longer and the retesting longer, all together. This script looks at the other extreme, where every task goes equally well or badly. Save it as together.py, with pert_sample copied into estimate_lib.py:

"""If every task goes well or badly together (one shared cause, like a late build),
the total's percentiles are the sums of each task's percentiles."""
import csv
import random
import sys

from estimate_lib import pert_sample

rows = list(csv.DictReader(open(sys.argv[1], encoding="utf-8")))
tasks = [(float(r["optimistic"]), float(r["likely"]), float(r["pessimistic"])) for r in rows]
random.seed(13)
RUNS = 100_000
per_task = [sorted(pert_sample(o, m, p) for _ in range(RUNS)) for o, m, p in tasks]
for q in (0.5, 0.8, 0.95):
    print(f"  {int(q * 100)}%: {sum(s[int(q * RUNS) - 1] for s in per_task):6.1f} hours")

tasks independent:
  50% chance of finishing within   81.0 hours
  80% chance of finishing within   86.6 hours
  95% chance of finishing within   92.1 hours
tasks move together:
  50%:   79.3 hours
  80%:   95.5 hours
  95%:  111.0 hours

The middle barely moves, but the 80% point rises from 86.6 to 95.5 hours, and the 95% point from 92 to 111. Real projects sit somewhere between these two cases. So when you give the 80% number, also say what would push it up: "86 hours, or up to 95 if the build arrives late or unstable."

Step 5: Check against your own history

The best correction for your estimates is your own past. If your team tracks estimated and actual testing time per feature, compare them. The history below is an example, written for this post. Use your own. Save it as history.csv:

feature,estimated_hours,actual_hours
Saved addresses,40,52
Promo codes,56,61
Order tracking map,72,104
Rate your rider,24,27
Split the bill,64,90
Order for later,48,58

And calibrate.py:

"""How far off were past test estimates? Use it to adjust the next one."""
import csv
import statistics
import sys

rows = list(csv.DictReader(open(sys.argv[1], encoding="utf-8")))
ratios = []
for r in rows:
    ratio = float(r["actual_hours"]) / float(r["estimated_hours"])
    ratios.append(ratio)
    print(f"{r['feature']:<22}{r['estimated_hours']:>6} est {r['actual_hours']:>6} actual   x{ratio:.2f}")
print(f"\nmedian actual/estimate: x{statistics.median(ratios):.2f}  (range x{min(ratios):.2f} to x{max(ratios):.2f})")
if len(sys.argv) > 2:
    est = float(sys.argv[2])
    print(f"a new estimate of {est:.0f} h, adjusted: {est * statistics.median(ratios):.0f} h "
          f"(range {est * min(ratios):.0f} to {est * max(ratios):.0f} h)")

$ python calibrate.py history.csv 73
Saved addresses           40 est     52 actual   x1.30
Promo codes               56 est     61 actual   x1.09
Order tracking map        72 est    104 actual   x1.44
Rate your rider           24 est     27 actual   x1.12
Split the bill            64 est     90 actual   x1.41
Order for later           48 est     58 actual   x1.21

median actual/estimate: x1.25  (range x1.09 to x1.44)
a new estimate of 73 h, adjusted: 92 h (range 80 to 105 h)

In this example, the team's past estimates were all too low, by 9% to 44%. Applied to the sum of most likely times, 73 hours becomes about 92. That's close to the simulation's 80% and 95% points, which is what you'd expect if the past estimates were also sums of most likely times. Two different methods pointing to the same range is a good sign. If they disagree badly, find out why before you give a number.

The biggest overruns in the example are the features with the most integration: a map and split payments. That's worth knowing for the next estimate too.

Step 6: "Can you do it in two weeks?"

The release plan has 10 working days for testing. At about 6 focused testing hours a day, that's 60 hours:

$ python estimate.py tasks.csv 60
...
  chance of finishing within 60 hours: 0.0%

None of the 100,000 simulated projects finished in 60 hours. So the honest answer to "can you do it in two weeks?" is no, not with this scope.

The usual next move is to cut scope. Say we automate only the 4 riskiest cases instead of 10, and run one exploratory session instead of three:

-Automate the 10 most important cases,12,16,30
+Automate the 4 riskiest cases,5,7,13
-Exploratory testing (3 sessions),5,6,9
+Exploratory testing (1 session),2,2,3

$ python estimate.py tasks-cut.csv 60
...
sum of PERT expected:        67.2 hours

simulated 100,000 projects:
  50% chance of finishing within   66.8 hours
  80% chance of finishing within   71.8 hours
  95% chance of finishing within   76.8 hours
  chance of finishing within the sum of 'most likely' (60 h): 9.3%
  mean 67.1 h, standard deviation 5.5 h
  chance of finishing within 60 hours: 9.3%

Look at that carefully. After the cut, the most likely times add up to exactly 60 hours. On a spreadsheet, the plan now fits. In the simulation, it fits 9.3% of the time. This is how most unrealistic test plans are made: cut until the most likely numbers add up to the deadline, and call it agreed.

What you can honestly offer instead:

• Time: "With the reduced scope, 72 hours, about 12 days, gets us to 80% confidence."
• More scope cut: say which further tests you'd drop, and what risk that leaves untested. That's a business decision, so let the business make it.
• Fewer unknowns: the widest range is in retesting. An earlier, more stable build narrows it more than any amount of overtime.
• More people: helps the tasks that can be split, like running test cases, but not reading the stories or the final report. Two people don't make it twice as fast.

How to say the number

An estimate is a forecast, so say it like one:

"Testing group ordering: about 87 hours, 15 working days, at 80% confidence. Up to 95 if the build is late or unstable. That includes two rounds of retesting. With 4 automated cases instead of 10 and one exploratory session, it's 72 hours, 12 days."

• A range or a confidence, not a single date.
• The assumptions: what's included, what the build needs to be like.
• The options if it's too long, with what each one costs.

Then re-estimate as you learn. After the first test run, you know how buggy the build is, and the retest range narrows. Tell people when your number changes, and why. An updated estimate is information, not failure.

Common mistakes

• Adding up the most likely times. In our example, that total was met 8.7% of the time.
• One line for "testing". Break it down, so each task can be checked and nothing is forgotten.
• No line for retesting and regression. It's often the biggest and most uncertain task.
• Eight-hour days. Use the hours you actually spend testing, usually 5 to 6.
• Cutting scope until the most likely times fit the deadline. That gave a 9.3% plan. Check the chance, not the sum.
• Never comparing estimates with actuals. Without history, the same mistake repeats on every feature.
• Treating the estimate as a promise. Re-estimate after the first test run, when you know much more.

What's next

In Part 14 of Testing in Practice we look at the test summary report and test metrics: what to put in the report at the end of testing, which numbers help people decide, and which ones only look good.

ProvaLab's test management system records how long every test result took to run and when each test run started and finished, so the history behind Step 5 builds up as you test instead of being reconstructed from memory. Test plans keep the scope and the dates together, so when scope is cut, the plan shows what was dropped. See it in the two-minute product tour.

Most summary reports fail that job the same way. They lead with a big number, such as "96.8% of tests passed", and the big number hides the one thing that matters.

This post builds a summary report from real test results. We ran the actual test suites from earlier parts of this series as one FoodNow release candidate, with five tools' results in one place. A short Python script turns the results into a report by requirement and risk. Every result and every line of output below is real. Two things were deliberate:

• We put one bug from Part 12 back into the release candidate.
• We wrote the list of requirements and their risks for this example. In a real project, they come from your test plan.

What goes in a test summary report

The order matters. Lead with what affects the decision, not with what's easiest to count.

Step 1: Collect the results in one format

Every test tool can write JUnit XML, a simple format that most CI systems and reporting tools read. We ran five suites from this series against the release candidate:

# Parts 8, 11, 7 and 9: pytest suites
pytest tests --junitxml=results/pricing-bdd.xml           # BDD pricing scenarios
pytest tests --junitxml=results/order-page-selenium.xml   # Selenium order page
pytest tests --junitxml=results/orders-contract.xml       # Pact contract
pytest tests --junitxml=results/order-for-later.xml       # order for later

# Part 12: Cypress, with its built-in JUnit reporter
npx cypress run --spec cypress/e2e/order.cy.js \
  --reporter junit --reporter-options "mochaFile=results/order-page-cypress.xml"

9 passed in 0.26s
4 passed in 14.59s
6 passed, 1 warning in 6.30s
1 failed, 4 passed, 1 warning in 1.88s
 │ Passing: 5 │
 │ Failing: 1 │

The Cypress failure we expected: it's the bug we put back. The order-for-later failure we didn't. We'll come to it in Step 4.

Step 2: The number everyone asks for

Save this as summary.py. It reads every JUnit file in a folder, and a list of requirements:

"""Build a test summary from JUnit XML files, by requirement and risk, not just by count."""
import csv
import glob
import re
import sys
import xml.etree.ElementTree as ET
from pathlib import Path

results_dir, req_file = sys.argv[1], sys.argv[2]
triage_file = sys.argv[3] if len(sys.argv) > 3 else None

# 1. Every test result, as "suite::test name".
tests = {}
for f in sorted(glob.glob(f"{results_dir}/*.xml")):
    suite = Path(f).stem
    for tc in ET.parse(f).iter("testcase"):
        problem = tc.find("failure")
        if problem is None:
            problem = tc.find("error")
        message = ""
        if problem is not None:
            message = (problem.get("message") or problem.text or "").strip().splitlines()[0][:90]
        tests[f"{suite}::{tc.get('name')}"] = ("failed" if problem is not None else "passed", message)

triage = {}
if triage_file:
    for row in csv.DictReader(open(triage_file, encoding="utf-8")):
        triage[row["test"]] = (row["kind"], row["note"])

total = len(tests)
passed = sum(1 for s, _ in tests.values() if s == "passed")
print(f"Tests run: {total}   passed: {passed}   failed: {total - passed}   pass rate: {passed / total:.1%}\n")

# 2. By requirement.
print(f"{'req':<4}{'risk':<8}{'tests':>6}{'failed':>8}  status       requirement")
blockers, mapped = [], set()
for r in csv.DictReader(open(req_file, encoding="utf-8")):
    ids = [t for t in tests if r["tests"] and re.search(r["tests"], t, flags=re.IGNORECASE)]
    mapped.update(ids)
    failed = [t for t in ids if tests[t][0] == "failed"]
    product_failures = [t for t in failed if triage.get(t, ("product bug",))[0] != "test bug"]
    if not ids:
        status = "NOT TESTED"
    elif product_failures:
        status = "FAILING"
    elif failed:
        status = "test issue"
    else:
        status = "passing"
    print(f"{r['id']:<4}{r['risk']:<8}{len(ids):>6}{len(failed):>8}  {status:<12} {r['requirement']}")
    if r["risk"] == "high" and status in ("NOT TESTED", "FAILING"):
        blockers.append(f"{r['id']} ({r['risk']} risk) {status.lower()}: {r['requirement']}")

unmapped = [t for t in tests if t not in mapped]
if unmapped:
    print(f"\n{len(unmapped)} test(s) not linked to any requirement:")
    for t in unmapped:
        print("  ", t)

# 3. Failures, with their triage.
print("\nFailures:")
for t, (status, msg) in tests.items():
    if status == "failed":
        kind, note = triage.get(t, ("not triaged", ""))
        print(f"  [{kind}] {t}\n      {msg}" + (f"\n      triage: {note}" if note else ""))

# 4. Against the exit criterion: every high-risk requirement tested, with no product failures.
print("\nReady to release?", "NO" if blockers else "YES")
for b in blockers:
    print("  -", b)

The first line of its output is the number most reports lead with:

Tests run: 30   passed: 28   failed: 2   pass rate: 93.3%

93.3% sounds nearly ready. It tells you nothing about what failed, or what was never tested.

Step 3: Link every test to a requirement and its risk

A release decision is about requirements, not tests. Which things that customers need are known to work, which are known to be broken, and which nobody checked? This requirements.csv lists FoodNow's requirements for this release, each with its risk and a pattern that picks out its tests:

id,requirement,risk,tests
R1,Totals and delivery fee are right,high,pricing-bdd::.*(delivery|big_order)|order-page-.*::.*(delivery|500)
R2,Promo codes work and are capped,medium,pricing-bdd::.*(first50|unknown_code|discount)
R3,The basket can be edited,medium,order-page-.*::.*(remov|empty)
R4,Customers are told when the price can't be worked out,high,order-page-cypress::.*price service
R5,Order for later offers only future slots in kitchen time,high,order-for-later::.*
R6,The app and the orders API agree,high,orders-contract::.*
R7,Group order: colleagues add dishes and one person pays,high,

R7 has no pattern, because no tests exist for it yet. Group ordering is in the release, but its tests weren't written in time.

Our first run of the script printed this:

R1  high         4       0  passing      Totals and delivery fee are right
R2  medium       0       0  NOT TESTED   Promo codes work and are capped
R3  medium       0       0  NOT TESTED   The basket can be edited
R4  high         0       0  NOT TESTED   Customers are told when the price can't be worked out
...
15 test(s) not linked to any requirement:
   order-page-cypress::FoodNow order charges delivery on a small order
   ...

Half the tests weren't linked to anything. That was a bug in our first version of the script: it used re.fullmatch, so a pattern had to match the whole test name. The fixed version above uses re.search. The lesson is the check itself: always list the tests that aren't linked to any requirement. Without that list, a mistake in the linking would quietly turn "tested" into "NOT TESTED", or the other way round, and the report would be wrong without anyone knowing.

With the fix, every test is linked:

$ python summary.py results requirements.csv
Tests run: 30   passed: 28   failed: 2   pass rate: 93.3%

req risk     tests  failed  status       requirement
R1  high         9       0  passing      Totals and delivery fee are right
R2  medium       4       0  passing      Promo codes work and are capped
R3  medium       4       0  passing      The basket can be edited
R4  high         2       1  FAILING      Customers are told when the price can't be worked out
R5  high         5       1  FAILING      Order for later offers only future slots in kitchen time
R6  high         6       0  passing      The app and the orders API agree
R7  high         0       0  NOT TESTED   Group order: colleagues add dishes and one person pays

Failures:
  [not triaged] order-for-later::test_page_offers_slots_from_now
      IndexError: list index out of range
  [not triaged] order-page-cypress::when the price service misbehaves says so when the price service is down
      Timed out retrying after 4000ms: expected '<span#total>' to contain text 'couldn\'t work o

Ready to release? NO
  - R4 (high risk) failing: Customers are told when the price can't be worked out
  - R5 (high risk) failing: Order for later offers only future slots in kitchen time
  - R7 (high risk) not tested: Group order: colleagues add dishes and one person pays

Now the picture is different. Two failures, each in a high-risk requirement, and a high-risk feature that's in the release with no tests at all.

Step 4: Triage every failure

A failed test means something went wrong. It doesn't yet say whether it's the product, the test, or the environment. Every failure needs a person to look at it before it goes into a report.

The Cypress failure is the bug we put back: when the price service fails, the page shows "Total: ₹undefined". A product bug.

The order-for-later failure we didn't plan. The test is from Part 9:

def test_page_offers_slots_from_now():
    html = client.get("/").text
    first = fn.slots(fn.now())[0]
    assert f"{first:%H:%M} –" in html

We ran the suites at 23:09. The kitchen closes at 23:00, so there are no slots, and [0] fails with IndexError. The page itself was right: it said the kitchen was closed. We checked exactly when this test fails:

time   slots  first
21:44  1      22:30
21:45  1      22:30
21:46  0      -
07:59  0      -
08:00  1      11:00

It fails from 21:46 to 07:59, every day: more than ten hours out of every twenty-four. When we wrote and ran it for Part 9, it was evening, and it passed. Anyone running the nightly build would have seen it fail every night. This is a test bug: the test depends on the real clock.

The triage decisions go in triage.csv, so the report shows them:

test,kind,note
order-for-later::test_page_offers_slots_from_now,test bug,"Fails from 21:46 to 07:59 every day: no slots are left, and the test reads slot [0]. The page correctly says the kitchen is closed."
order-page-cypress::when the price service misbehaves says so when the price service is down,product bug,"Regression of the Part 12 fix: the page shows 'Total: ₹undefined' when the price service returns 500."

Step 5: Fix the test, and run the report again

The fix is to give the test a fixed time instead of the real clock, and to add a test for the closed kitchen, which is real behaviour worth checking:

def test_page_offers_slots_from_now(monkeypatch):
    monkeypatch.setattr(fn, "now", lambda: datetime(2026, 11, 11, 19, 10))
    html = client.get("/").text
    assert "20:00 – 20:30" in html


def test_page_says_when_the_kitchen_is_closed(monkeypatch):
    monkeypatch.setattr(fn, "now", lambda: datetime(2026, 11, 11, 23, 9))
    assert "The kitchen is closed" in client.get("/").text

6 passed, 1 warning in 0.51s

And the report, with the triage:

$ python summary.py results requirements.csv triage.csv
Tests run: 31   passed: 30   failed: 1   pass rate: 96.8%

req risk     tests  failed  status       requirement
R1  high         9       0  passing      Totals and delivery fee are right
R2  medium       4       0  passing      Promo codes work and are capped
R3  medium       4       0  passing      The basket can be edited
R4  high         2       1  FAILING      Customers are told when the price can't be worked out
R5  high         6       0  passing      Order for later offers only future slots in kitchen time
R6  high         6       0  passing      The app and the orders API agree
R7  high         0       0  NOT TESTED   Group order: colleagues add dishes and one person pays

Failures:
  [product bug] order-page-cypress::when the price service misbehaves says so when the price service is down
      Timed out retrying after 4000ms: expected '<span#total>' to contain text 'couldn\'t work o
      triage: Regression of the Part 12 fix: the page shows 'Total: ₹undefined' when the price service returns 500.

Ready to release? NO
  - R4 (high risk) failing: Customers are told when the price can't be worked out
  - R7 (high risk) not tested: Group order: colleagues add dishes and one person pays

The pass rate went up, from 93.3% to 96.8%. The answer is still no. One high-risk requirement fails, and one isn't tested at all. Notice also that R7 doesn't appear anywhere in the pass rate: a feature with no tests can't fail a test. That's why "what wasn't tested" needs its own section.

Step 6: Write the report

The script gives the facts. The report adds what they mean and what you recommend. Here's the report we'd write for this release candidate. The build name, dates and fix times are made up for the example; the results are the real ones above:

FoodNow release 26.11, test summary

Tested: build 26.11-rc2 on the staging environment, 10–11 November. Five suites, 31 automated tests, linked to 7 requirements.

Recommendation: don't release yet. Two high-risk problems:

1. When the price service fails, customers see "Total: ₹undefined" and can still place the order (R4). This is a regression: it was fixed in the previous release. Fix and retest: about half a day.
2. Group ordering (R7) has no tests. It's new in this release, and it handles payment for several people. We don't know whether it works. Testing it needs about 3 days, or the feature can be switched off for this release.

Working: totals and delivery fees, promo codes, basket editing, order for later, and the app–API contract all pass (29 tests).

Fixed during testing: one test failed every night between 21:46 and 07:59 because it used the real clock. It's been fixed, and a test for the closed kitchen was added.

If we release anyway: customers whose price check fails will see a broken total, and group orders go out untested.

Short, specific and honest about what's not known. Someone who reads only the first three lines knows what to decide.

Metrics: which ones help

A warning that applies to every metric: when a measure becomes a target, people improve the measure instead of the thing. This is known as Goodhart's law. Make a pass rate a target, and tests that fail get deleted or skipped. Use metrics to ask questions, not to grade people.

Common mistakes

• Leading with the pass rate. Lead with the recommendation and the high-risk problems.
• Not reporting what wasn't tested. An untested feature never shows up in a pass rate. Give it its own line.
• Reporting failures without triage. A test bug, an environment problem and a product bug look the same in a raw count.
• No link from tests to requirements. Without it, you can't say which customer needs are safe, only how many tests passed.
• Not checking the linking. List the unlinked tests. Our first run had 15.
• Tests that depend on the real clock. Ours failed for 10 hours a day. Give tests a fixed time.
• Writing for testers. The reader is deciding whether to release. Say what each problem means for a customer.

What's next

In Part 15 of Testing in Practice we look at localization testing: the FoodNow app in Hindi, German and Arabic, with Playwright switching languages and number formats, and the bugs that only appear when the text isn't in English.

ProvaLab's test management system builds this kind of report from your test runs without a script. Requirements are linked to the test cases that check them, the requirement coverage view shows which ones pass, fail or have no tests, and issues are linked to the failing results with their severity. Results from pytest, Cypress, Playwright and other frameworks arrive through ProvaLab's plugins, so five tools end up in one report. See it in the two-minute product tour.

Database testing checks what is actually stored after your code runs. This post builds a small order system for FoodNow, our food delivery example, on a real PostgreSQL database. It then tests it with pytest and plain SQL.

Every command and every line of output below was run while writing this post. Two bugs are planted in the code. The usual tests pass with both of them in. The database tests catch both, and a third bug that lives in a migration.

What to check when code writes to a database

The first one is what most people test. The bugs are usually in the other five.

Step 1: Set up (5 minutes)

You need Docker and Python 3.10 or newer. Start PostgreSQL and make an empty database for the tests:

docker run -d --name foodnow-db -e POSTGRES_PASSWORD=postgres -p 5432:5432 postgres:15
docker exec foodnow-db createdb -U postgres foodnow_test

mkdir foodnow-db && cd foodnow-db
python -m venv .venv && source .venv/bin/activate
pip install "psycopg[binary]==3.3.8" pytest==9.2.0

We used PostgreSQL 15, Python 3.11, psycopg 3.3.8 and pytest 9.2.0 on Linux.

The schema, schema.sql, has three tables: the menu, the orders, and which dishes are in each order. Prices are in whole rupees. The CHECK rules match the business rules: the tip is 0 to 200, and stock can't go below zero.

CREATE TABLE menu_items (
    id     serial PRIMARY KEY,
    name   text    NOT NULL,
    price  integer NOT NULL CHECK (price >= 0),
    stock  integer NOT NULL CHECK (stock >= 0)
);

CREATE TABLE orders (
    id         serial PRIMARY KEY,
    customer   text    NOT NULL,
    tip        integer NOT NULL DEFAULT 0 CHECK (tip BETWEEN 0 AND 200),
    total      integer NOT NULL,
    created_at timestamptz NOT NULL DEFAULT now()
);

CREATE TABLE order_items (
    order_id integer NOT NULL REFERENCES orders(id),
    item_id  integer NOT NULL REFERENCES menu_items(id),
    qty      integer NOT NULL CHECK (qty > 0),
    PRIMARY KEY (order_id, item_id)
);

Step 2: The code under test

orders.py places an order: work out the total, save the order, then take each dish out of stock and save it as an order line. If a dish has run out, it raises OutOfStock, and the app shows "Sorry, out of stock".

class OutOfStock(Exception):
    pass


def place_order(conn, customer, items, tip=0):
    """items is {menu_item_id: quantity}. Returns the new order id."""
    total = tip
    for item_id, qty in items.items():
        price = conn.execute(
            "SELECT price FROM menu_items WHERE id = %s", (item_id,)
        ).fetchone()[0]
        total += price * qty

    order_id = conn.execute(
        "INSERT INTO orders (customer, tip, total) VALUES (%s, %s, %s) RETURNING id",
        (customer, tip, total),
    ).fetchone()[0]

    for item_id, qty in items.items():
        stock = conn.execute(
            "SELECT stock FROM menu_items WHERE id = %s", (item_id,)
        ).fetchone()[0]
        if stock < qty:
            raise OutOfStock(f"item {item_id}: {stock} left, {qty} wanted")
        conn.execute(
            "UPDATE menu_items SET stock = %s WHERE id = %s", (stock - qty, item_id)
        )
        conn.execute(
            "INSERT INTO order_items (order_id, item_id, qty) VALUES (%s, %s, %s)",
            (order_id, item_id, qty),
        )
    return order_id

It reads well, and it has two bugs. See if you can spot them before the tests do.

Step 3: A fresh database for every test

Tests that share a database start depending on each other: one test leaves a row behind, and another one fails because of it, but only when they run in a certain order. The fix is to give every test a clean start. This fixture, in tests/conftest.py, drops everything, creates the schema again, and puts three dishes on the menu:

import os
import pathlib

import psycopg
import pytest

DB_URL = os.environ.get("DB_URL", "postgresql://postgres:postgres@localhost:5432/foodnow_test")
SCHEMA = (pathlib.Path(__file__).parent.parent / "schema.sql").read_text()


@pytest.fixture
def db():
    """A fresh, empty schema for every test, with three dishes on the menu."""
    with psycopg.connect(DB_URL, autocommit=True) as conn:
        conn.execute("DROP SCHEMA public CASCADE; CREATE SCHEMA public;")
        conn.execute(SCHEMA)
        conn.execute(
            "INSERT INTO menu_items (name, price, stock) VALUES "
            "('Paneer wrap', 250, 10), ('Masala dosa', 180, 10), ('Mango lassi', 90, 1)"
        )
        yield conn

Creating the schema again takes a few milliseconds here. With hundreds of tables, teams often start each test inside a transaction and roll it back at the end instead, which is faster. That doesn't work for the race test in Step 6, though, because there other connections need to see committed data. We use the simple version for everything.

Use the same database as production. It's tempting to test with SQLite because it needs no server. But SQLite behaves differently: for example, it doesn't enforce foreign keys unless you turn them on with PRAGMA foreign_keys = ON, and its locking is nothing like PostgreSQL's. The race in Step 6 is about exactly that. A test on a different database tests a different program.

Step 4: The usual tests pass

These are the tests most teams write first. The happy path checks the stored order, the stored lines and the stock, which is already better than checking only the return value. The second test checks that an order for too many lassis is refused.

import pytest

from orders import OutOfStock, place_order

WRAP, DOSA, LASSI = 1, 2, 3


def test_order_is_saved_with_the_right_total(db):
    order_id = place_order(db, "asha", {WRAP: 2}, tip=50)

    row = db.execute("SELECT customer, tip, total FROM orders WHERE id = %s", (order_id,)).fetchone()
    assert row == ("asha", 50, 550)
    items = db.execute("SELECT item_id, qty FROM order_items WHERE order_id = %s", (order_id,)).fetchall()
    assert items == [(WRAP, 2)]
    assert db.execute("SELECT stock FROM menu_items WHERE id = %s", (WRAP,)).fetchone()[0] == 8


def test_out_of_stock_order_is_refused(db):
    with pytest.raises(OutOfStock):
        place_order(db, "ravi", {LASSI: 2})

$ python -m pytest -q tests
..                                                                       [100%]
2 passed in 0.14s

Green. And the second test is right: the order is refused. It just doesn't ask what the refusal left behind.

Step 5: Bug 1, the order that was refused and saved anyway

Ravi orders a wrap and two lassis. There is only one lassi, so he gets "Sorry, out of stock". We ran exactly that, after Asha's order from Step 4, and then looked in the database:

 id | customer | tip | total |          created_at
----+----------+-----+-------+-------------------------------
  1 | asha     |  50 |   550 | 2026-11-10 17:47:06.680316+00
  2 | ravi     |   0 |   430 | 2026-11-10 17:47:06.69123+00

 order_id | item_id | qty
----------+---------+-----
        1 |       1 |   2
        2 |       1 |   1

 id |    name     | price | stock
----+-------------+-------+-------
  1 | Paneer wrap |   250 |     7
  2 | Masala dosa |   180 |    10
  3 | Mango lassi |    90 |     1

Ravi was told his order failed, but order 2 is in the database. It has a total of 430 (one wrap plus two lassis) and only one line, a wrap, and the wrap was taken out of stock. The kitchen may cook it. A report of the day's sales will count 430 rupees nobody paid.

The cause: psycopg's autocommit=True (Django's default mode works the same way) saves each statement the moment it runs. The order row and the wrap were saved before the lassi check failed, and nothing undid them.

The test that catches it checks what the refusal left behind:

def test_refused_order_leaves_no_trace(db):
    with pytest.raises(OutOfStock):
        place_order(db, "ravi", {WRAP: 1, LASSI: 2})

    assert db.execute("SELECT count(*) FROM orders").fetchone()[0] == 0
    assert db.execute("SELECT count(*) FROM order_items").fetchone()[0] == 0
    assert db.execute("SELECT stock FROM menu_items WHERE id = %s", (WRAP,)).fetchone()[0] == 10

The same bug, found with SQL

Tests catch it before release. But what if it already happened in production? You can look for the damage with a query. This one, checks.sql, finds every order whose total doesn't match its lines plus the tip:

SELECT o.id, o.customer, o.total,
       o.tip + coalesce(sum(m.price * oi.qty), 0) AS items_plus_tip
FROM orders o
LEFT JOIN order_items oi ON oi.order_id = o.id
LEFT JOIN menu_items m ON m.id = oi.item_id
GROUP BY o.id
HAVING o.total <> o.tip + coalesce(sum(m.price * oi.qty), 0);

 id | customer | total | items_plus_tip
----+----------+-------+----------------
  2 | ravi     |   430 |            250
(1 row)

Queries like this are worth keeping. Run them after a data migration, or every night against a copy of production: they find broken data no matter which bug caused it. (This one uses today's prices, so in a real system you would store the price on each order line.)

Step 6: Bug 2, twenty customers and ten wraps

There are 10 paneer wraps. At lunchtime 20 customers press Pay at the same moment, each for one wrap. Ten should get their wrap and ten should get "out of stock".

A test can do that: 20 threads, each with its own database connection, held at a barrier so they all start together.

import threading

import psycopg

from orders import OutOfStock, place_order
from conftest import DB_URL

WRAP = 1  # 10 in stock


def test_twenty_customers_cannot_buy_ten_wraps(db):
    results = []
    start = threading.Barrier(20)

    def customer(n):
        with psycopg.connect(DB_URL, autocommit=True) as conn:
            start.wait()  # everyone clicks Pay at the same moment
            try:
                place_order(conn, f"customer{n}", {WRAP: 1})
                results.append("ok")
            except OutOfStock:
                results.append("out of stock")

    threads = [threading.Thread(target=customer, args=(n,)) for n in range(20)]
    for t in threads:
        t.start()
    for t in threads:
        t.join()

    sold = db.execute("SELECT coalesce(sum(qty), 0) FROM order_items WHERE item_id = %s", (WRAP,)).fetchone()[0]
    stock = db.execute("SELECT stock FROM menu_items WHERE id = %s", (WRAP,)).fetchone()[0]
    # (orders accepted, wraps sold, wraps left)
    assert (results.count("ok"), sold, stock) == (10, 10, 0)

Here is the full run, with the tests from Steps 5 and 6 added (the schema and migration tests are in Steps 7 and 8; this run already had the fixed migration from Step 8):

$ python -m pytest tests --tb=short
============================= test session starts ==============================
platform linux -- Python 3.11.17, pytest-9.2.0, pluggy-1.7.0
rootdir: ~/foodnow-db
collected 8 items

tests/test_concurrency.py F                                              [ 12%]
tests/test_migrations.py ..                                              [ 37%]
tests/test_orders.py ..F                                                 [ 75%]
tests/test_schema.py ..                                                  [100%]

=================================== FAILURES ===================================
__________________ test_twenty_customers_cannot_buy_ten_wraps __________________
tests/test_concurrency.py:33: in test_twenty_customers_cannot_buy_ten_wraps
    assert (results.count("ok"), sold, stock) == (10, 10, 0)
E   assert (20, 20, 9) == (10, 10, 0)
E
E     At index 0 diff: 20 != 10
E     Use -v to get more diff
______________________ test_refused_order_leaves_no_trace ______________________
tests/test_orders.py:27: in test_refused_order_leaves_no_trace
    assert db.execute("SELECT count(*) FROM orders").fetchone()[0] == 0
E   assert 1 == 0
=========================== short test summary info ============================
FAILED tests/test_concurrency.py::test_twenty_customers_cannot_buy_ten_wraps
FAILED tests/test_orders.py::test_refused_order_leaves_no_trace - assert 1 == 0
========================= 2 failed, 6 passed in 0.50s ==========================

All 20 orders were accepted. 20 wraps were sold. And the stock still says 9. We ran it five more times: 20 accepted every time, with 8 or 9 left in stock. Not one customer got "out of stock".

The CHECK (stock >= 0) rule didn't help, because the stock never went below zero. Each customer read the stock (10), worked out the new number in Python (9), and wrote that number back. Most of them read 10 before anyone had written, so they all wrote 9. This is called a lost update: every write is based on an old read and overwrites the others.

None of the single-user tests in Step 4 can find this. A manual tester clicking Pay can't find it either. It only shows up with many requests at once: at lunchtime, or when a page retries a slow request.

The fix, and one that doesn't work

We tried wrapping the old code in a transaction first. That alone doesn't fix the race: we ran it five times and got 20 orders accepted and 9 left every time. With PostgreSQL's default isolation level, Read Committed, two transactions can still both read 10 before either writes.

What fixes both bugs is two changes:

def place_order(conn, customer, items, tip=0):
    """items is {menu_item_id: quantity}. Returns the new order id."""
    with conn.transaction():  # all of it, or none of it
        total = tip
        for item_id, qty in items.items():
            price = conn.execute(
                "SELECT price FROM menu_items WHERE id = %s", (item_id,)
            ).fetchone()[0]
            total += price * qty

        order_id = conn.execute(
            "INSERT INTO orders (customer, tip, total) VALUES (%s, %s, %s) RETURNING id",
            (customer, tip, total),
        ).fetchone()[0]

        for item_id, qty in items.items():
            # Check and take the stock in one statement, so no one can sneak in between.
            taken = conn.execute(
                "UPDATE menu_items SET stock = stock - %s "
                "WHERE id = %s AND stock >= %s RETURNING stock",
                (qty, item_id, qty),
            ).fetchone()
            if taken is None:
                raise OutOfStock(f"item {item_id}: not enough left for {qty}")
            conn.execute(
                "INSERT INTO order_items (order_id, item_id, qty) VALUES (%s, %s, %s)",
                (order_id, item_id, qty),
            )
    return order_id

1. with conn.transaction(): makes the whole order one unit. If OutOfStock is raised, psycopg rolls everything back: the order row, the lines, the stock. That fixes bug 1.
2. One UPDATE that checks and takes the stock. stock = stock - 1 WHERE stock >= 1 lets the database do the arithmetic on the current value. When two customers run it at once, PostgreSQL makes the second wait until the first is finished, then checks the WHERE again on the new value. If nothing is left, no row matches, RETURNING returns nothing, and the code raises OutOfStock. That fixes bug 2. (SELECT ... FOR UPDATE before the read also works. It locks the row while you decide.)

$ python -m pytest tests
============================= test session starts ==============================
platform linux -- Python 3.11.17, pytest-9.2.0, pluggy-1.7.0
rootdir: ~/foodnow-db
collected 8 items

tests/test_concurrency.py .                                              [ 12%]
tests/test_migrations.py ..                                              [ 37%]
tests/test_orders.py ...                                                 [ 75%]
tests/test_schema.py ..                                                  [100%]

============================== 8 passed in 0.45s ===============================

We ran the fixed suite 10 times: 8 passed every time. And the 20-customer race, run 5 more times on its own: exactly 10 accepted, 10 sold, 0 left, every time.

A race test that passes once proves little, because races depend on timing. Run it many times, for example for i in $(seq 20); do python -m pytest -q tests/test_concurrency.py; done, and keep it in your suite, where it runs on every change.

Step 7: Test the constraints too

The CHECK rules in the schema are the last line of defence: they protect the data even when some other code, a script or a person in a SQL console, skips the rules in your Python. They are worth two lines of test each, because a migration can quietly drop them.

import psycopg
import pytest


def test_database_refuses_a_tip_over_200(db):
    with pytest.raises(psycopg.errors.CheckViolation):
        db.execute("INSERT INTO orders (customer, tip, total) VALUES ('asha', 250, 750)")


def test_database_refuses_negative_stock(db):
    with pytest.raises(psycopg.errors.CheckViolation):
        db.execute("UPDATE menu_items SET stock = -1 WHERE id = 1")

Step 8: The migration that only fails on real data

FoodNow adds a delivery fee. The migration, migrations/002_delivery_fee.sql, is one line:

ALTER TABLE orders ADD COLUMN delivery_fee integer NOT NULL;

The developer runs it on their test database, which is empty, and it works. Production has a year of orders. Two tests, one for each case:

import pathlib

MIGRATION = (pathlib.Path(__file__).parent.parent / "migrations/002_delivery_fee.sql").read_text()


def test_migration_on_an_empty_database(db):
    db.execute(MIGRATION)


def test_migration_on_a_database_that_already_has_orders(db):
    db.execute("INSERT INTO orders (customer, tip, total) VALUES ('asha', 50, 550)")

    db.execute(MIGRATION)

    row = db.execute("SELECT customer, total, delivery_fee FROM orders").fetchone()
    assert row == ("asha", 550, 0)

tests/test_migrations.py .F                                              [100%]

=================================== FAILURES ===================================
_____________ test_migration_on_a_database_that_already_has_orders _____________
tests/test_migrations.py:13: in test_migration_on_a_database_that_already_has_orders
    db.execute(MIGRATION)
.../site-packages/psycopg/connection.py:304: in execute
    raise ex.with_traceback(None)
E   psycopg.errors.NotNullViolation: column "delivery_fee" of relation "orders" contains null values
=========================== short test summary info ============================
FAILED tests/test_migrations.py::test_migration_on_a_database_that_already_has_orders
========================= 1 failed, 1 passed in 0.12s ==========================

(We shortened the long path to the installed psycopg package to .../site-packages.)

On an empty table it passes. With one existing order it fails, because the old rows would get NULL in a column that doesn't allow it. In production, that's a failed release in the middle of a deploy. The fix is to give old rows a value:

ALTER TABLE orders ADD COLUMN delivery_fee integer NOT NULL DEFAULT 0;

Both tests pass. Since PostgreSQL 11, adding a column with a fixed default like this doesn't rewrite the whole table, so it's quick even on a big one.

Test every migration on data shaped like production: existing rows, NULLs in optional columns, old values that today's code would never write. The strongest version is a nightly job that runs the new migrations on a recent anonymised copy of production.

Common mistakes

• Checking only the return value. The function's answer and the database's contents can disagree, as Step 5 shows. After a write, query the rows. After a failure, query them too.
• Mocking the database in these tests. A mock returns what you told it to. It has no transactions, no locks and no constraints, which is where all three bugs here lived. Mocks have their place, but not in a test whose job is to check the database.
• Tests that share data. Shared rows make tests pass or fail depending on order. Give each test a fresh schema, or a transaction that is rolled back.
• Testing concurrency by hand. Two people can't click at exactly the same moment. Threads with a barrier, as in Step 6, can, and they do it on every run.
• Testing migrations only on an empty database. Empty tables accept almost any migration. Real tables are where they fail.

What's next

In Part 5 of Testing in Practice we look at test doubles: mocks, stubs, fakes and spies, what each one is for, and how a mock can let a real bug through.

ProvaLab's pytest plugin sends every result from tests like these to the ProvaLab test management system, with the history of each test across runs. So a race test that fails once in fifty runs shows up as a pattern you can look at, not a one-off you forget. See it in the two-minute product tour.

The ISO standard for usability, ISO 9241-11, defines it with three measures:

• Effectiveness: can people complete the task?
• Efficiency: how much time and effort does it take?
• Satisfaction: how do they feel about it?

This post shows the two main ways to find usability problems, on a FoodNow add a delivery address form for phones:

• A heuristic review: an expert checks the design against well-known rules. We did this for real, with Playwright filling in the form on a phone-sized screen and measuring what can be measured.
• A usability test with users: you watch real people try real tasks. That needs real people. We show how to plan and run one. The questionnaire answers in Step 6 are an example, and we say so where they appear.

Usability testing vs UAT

Part 9 was user acceptance testing. They're often confused:

Usability testing is cheapest early, on a sketch or prototype, when changing the design costs nothing. By UAT, the same finding costs a release.

The form

The add-address form, version 1. It's a small FastAPI app that renders this HTML:

<h1>Add address</h1>
<form method="post" action="/address">
  <a class="loc" href="#">Use current location</a>
  <input name="flat" placeholder="Flat / house no.">
  <input name="street" placeholder="Street and area">
  <input name="pincode" placeholder="Pincode">
  <input name="phone" placeholder="Phone for the rider">
  <div class="row">
    <button class="save">Save</button>
    <button class="cancel" type="button">Cancel</button>
  </div>
</form>

And the server-side check:

@app.post("/address", response_class=HTMLResponse)
def save(flat: str = Form(""), street: str = Form(""), pincode: str = Form(""), phone: str = Form("")):
    if not (flat and street and re.fullmatch(r"\d{6}", pincode) and re.fullmatch(r"\d{10}", phone)):
        return form("Error: invalid input")
    return page(f'<h1>Address saved</h1><p id="saved">{flat}, {street}, {pincode}</p>')

Every functional test you'd write for this passes: valid input saves, invalid input is refused. Now look at it as a customer would.

Step 1: Fill it in like a customer

This Playwright script opens the form on a phone-sized screen and fills it in the way a person in India would, including the phone number written the usual way, with the country code and a space. Save it as review.py (you need pip install playwright==1.64.0 and playwright install chromium):

import sys

from playwright.sync_api import sync_playwright

URL, SHOTS = sys.argv[1], sys.argv[2]

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_context(viewport={"width": 412, "height": 915}, device_scale_factor=2,
                               is_mobile=True, has_touch=True, locale="en-IN").new_page()
    page.goto(URL)

    # 1. Fill it in as a customer would, including the phone number as people write it.
    fields = page.locator("input")
    values = ["Flat 4B, Lotus Apartments", "14th Cross, Indiranagar", "560038", "+91 98765 43210"]
    for f, v in zip(fields.all(), values):
        f.fill(v)
    page.screenshot(path=f"{SHOTS}/2-filled.png")
    page.get_by_role("button", name="Save").click()
    page.wait_for_load_state()
    page.screenshot(path=f"{SHOTS}/3-after-save.png")
    print("after Save:", page.locator("h1").inner_text(), "|",
          page.locator(".err, [role=alert], #saved").all_inner_texts())
    print("values kept:", [f.input_value() for f in page.locator("input").all()])

    # 2. Labels and keyboards.
    page.goto(URL)
    for f in page.locator("input").all():
        label = f.evaluate("e => e.labels.length ? e.labels[0].innerText : null")
        print(f"field {f.get_attribute('name')!r}: label={label!r} placeholder={f.get_attribute('placeholder')!r} "
              f"type={f.get_attribute('type')!r} inputmode={f.get_attribute('inputmode')!r} "
              f"autocomplete={f.get_attribute('autocomplete')!r}")

    # 3. Tap targets: WCAG 2.2 asks for at least 24x24 CSS px; Android and iOS guides say 48 and 44.
    for el in page.locator("a, button, input").all():
        box = el.bounding_box()
        name = (el.inner_text() or el.get_attribute("name") or "").strip()
        flag = "  < 24px" if min(box["width"], box["height"]) < 24 else ("  < 44px" if min(box["width"], box["height"]) < 44 else "")
        print(f"target {name!r}: {box['width']:.0f} x {box['height']:.0f}{flag}")
    browser.close()

The output, on version 1:

after Save: Add address | ['Error: invalid input']
values kept: ['', '', '', '']
field 'flat': label=None placeholder='Flat / house no.' type=None inputmode=None autocomplete=None
field 'street': label=None placeholder='Street and area' type=None inputmode=None autocomplete=None
field 'pincode': label=None placeholder='Pincode' type=None inputmode=None autocomplete=None
field 'phone': label=None placeholder='Phone for the rider' type=None inputmode=None autocomplete=None
target 'Use current location': 114 x 17  < 24px
target 'flat': 372 x 44
target 'street': 372 x 44
target 'pincode': 372 x 44
target 'phone': 372 x 44
target 'Save': 62 x 34  < 44px
target 'Cancel': 300 x 53

The address was correct. The phone number was correct too, written the way most people in India write it. The form said "Error: invalid input" and emptied all four fields. A customer on a phone has to type the whole address again, without knowing what was wrong. Many won't. They'll order somewhere else.

A functional test with 9876543210 passes. Only a test that uses the input real people type finds this.

Step 2: The heuristic review

A heuristic review (or heuristic evaluation) checks a design against a short list of rules of thumb. The best known are Jakob Nielsen's 10 usability heuristics, first published in 1994. Two or three reviewers check the design separately, then combine their lists. It's quick and cheap, and it needs no users.

What we found on version 1, using the run above and the screenshots:

We used three severity levels: critical stops people from finishing, major causes real trouble or errors, minor slows them down or annoys them. (Nielsen's own scale has five, from 0 to 4, but the idea is the same.)

Notice that several of these are also accessibility failures. A placeholder disappears as soon as you type, so anyone who loses track has nothing to go back to, and WCAG asks for visible labels (success criterion 3.3.2). A 17-pixel link is hard to tap for anyone with shaky hands. Good usability and accessibility overlap a lot.

Step 3: Fix, and run the same checks

Version 2 changes the form, not the business rules:

FIELDS = [  # name, label, hint, extra attributes
    ("flat", "Flat or house number", "", 'autocomplete="address-line1"'),
    ("street", "Street and area", "", 'autocomplete="address-line2"'),
    ("pincode", "Pincode", "6 digits", 'inputmode="numeric" autocomplete="postal-code" maxlength="6"'),
    ("phone", "Phone for the rider", "10-digit mobile number", 'type="tel" autocomplete="tel"'),
]


def phone_digits(raw: str) -> str:
    d = re.sub(r"\D", "", raw)
    if len(d) == 12 and d.startswith("91"):
        d = d[2:]
    elif len(d) == 11 and d.startswith("0"):
        d = d[1:]
    return d

• Visible labels and hints above every field, linked with for and aria-describedby.
• Accept the formats people use. Spaces, +91 and a leading 0 are removed before the 10-digit check.
• Keep what was typed. On an error, the form comes back with every value.
• Say what's wrong, next to the field. "A pincode has 6 digits. This one has 5." And a summary at the top with role="alert", so screen readers announce it.
• One clear primary action. "Save address" is full width and bright. Cancel is a plain link below it.
• Right keyboards. inputmode="numeric" for the pincode, type="tel" for the phone, and autocomplete so the browser can fill the address.
• Tap targets of at least 48 pixels.

The same script, on version 2:

after Save: Address saved | ['Flat 4B, Lotus Apartments, 14th Cross, Indiranagar, 560038. The rider will call 9876543210.']
values kept: []
field 'flat': label='Flat or house number' placeholder=None type=None inputmode=None autocomplete='address-line1'
field 'street': label='Street and area' placeholder=None type=None inputmode=None autocomplete='address-line2'
field 'pincode': label='Pincode' placeholder=None type=None inputmode='numeric' autocomplete='postal-code'
field 'phone': label='Phone for the rider' placeholder=None type='tel' inputmode=None autocomplete='tel'
target 'Use my current location': 372 x 48
target 'flat': 372 x 48
target 'street': 372 x 48
target 'pincode': 372 x 48
target 'phone': 372 x 48
target 'Save address': 372 x 52
target 'Cancel': 372 x 48

The address saves (values kept is empty because the page now shows the confirmation instead of the form). And with a real mistake, a 5-digit pincode:

alert: Please check 1 field below.
pincode error: A pincode has 6 digits. This one has 5.
values kept: ['Flat 4B, Lotus Apartments', '14th Cross, Indiranagar', '56003', '+91 98765 43210']

Step 4: Why test with users at all?

A heuristic review found eight problems in a few minutes. Why go further?

Because reviewers aren't users. They know what a pincode is, they read every word, and they don't use the app on a crowded bus with one hand. Users find problems no rule predicts: a word they don't understand, a step they expected and didn't get, a field they fill with something nobody imagined. In our form, a user might type their building name in the flat field and leave the street empty, or expect "Use my current location" to fill everything. You only learn that by watching.

So use both. The review clears the obvious problems cheaply, so the users' time is spent on the ones only they can find.

Step 5: How many users? Usually five, then five more

The famous answer is five. It comes from Jakob Nielsen and Tom Landauer's 1993 research: if a usability problem affects a share L of users, the chance that at least one of n test users runs into it is 1 − (1 − L)ⁿ. In their projects, a typical problem affected about 31% of users. This script works it out:

"""Share of usability problems found by n test users: 1 - (1 - L)^n."""
for L, label in [(0.31, "a typical problem (L = 31%)"), (0.10, "a problem 1 user in 10 hits")]:
    print(label)
    for n in (1, 2, 3, 5, 8, 15):
        print(f"  {n:>2} users: {1 - (1 - L) ** n:6.1%}")

a typical problem (L = 31%)
   1 users:  31.0%
   2 users:  52.4%
   3 users:  67.1%
   5 users:  84.4%
   8 users:  94.9%
  15 users:  99.6%
a problem 1 user in 10 hits
   1 users:  10.0%
   2 users:  19.0%
   3 users:  27.1%
   5 users:  41.0%
   8 users:  57.0%
  15 users:  79.4%

Five users find about 84% of the common problems. But a problem that only 1 user in 10 hits is found just 41% of the time. That's why the advice is five users, fix, then five more, not "five users and done". Each round finds the problems that are left. And if you have very different groups of users, such as customers and delivery riders, test about five from each group.

Step 6: Run the sessions

A usability session takes 30 to 60 minutes, one user at a time:

1. Welcome them, and say what's being tested. "We're testing the app, not you. If something's confusing, that's what we need to know."
2. Give a realistic task, not instructions. "You've just moved to a new flat in Indiranagar. Add it as your delivery address." Not "tap Add address, then type in the first box".
3. Ask them to think aloud, and stay quiet. Don't help, don't explain, don't defend the design. If they ask, say "what would you do if I weren't here?"
4. Watch what they do, not only what they say. Hesitations, wrong taps, re-typing, looking for something.
5. Measure each task: completed or not, time taken, and errors.
6. End with a short questionnaire, usually the System Usability Scale.

The System Usability Scale (SUS), created by John Brooke in 1986, is ten statements that users rate from 1 (strongly disagree) to 5 (strongly agree), such as "I thought the system was easy to use" and "I found the system unnecessarily complex". Odd-numbered statements are positive and even-numbered ones are negative, and the score runs from 0 to 100. This script scores it:

"""Score the System Usability Scale (SUS): ten statements, each answered 1 (strongly disagree) to 5 (strongly agree)."""
import csv
import sys


def sus(answers: list[int]) -> float:
    assert len(answers) == 10 and all(1 <= a <= 5 for a in answers)
    odd = sum(a - 1 for a in answers[0::2])   # statements 1,3,5,7,9 are positive
    even = sum(5 - a for a in answers[1::2])  # statements 2,4,6,8,10 are negative
    return (odd + even) * 2.5


rows = list(csv.reader(open(sys.argv[1])))[1:]
scores = []
for name, *answers in rows:
    s = sus([int(a) for a in answers])
    scores.append(s)
    print(f"{name:<14} {s:5.1f}")
print(f"{'average':<14} {sum(scores) / len(scores):5.1f}  (68 is about average across many studies)")

The answers below are an example, written for this post to show how the scoring works. They are not from real users. Five people on version 1, and five different people on version 2:

--- v1 (example answers)
P1              55.0
P2              70.0
P3              35.0
P4              60.0
P5              75.0
average         59.0  (68 is about average across many studies)
--- v2 (example answers)
P6              82.5
P7              87.5
P8              72.5
P9              90.0
P10             75.0
average         81.5  (68 is about average across many studies)

Two things about SUS. First, the score is not a percentage: 68 is roughly average, based on hundreds of published studies, so 59 is below average, not "59% usable". Second, with five people per version the spread is wide (35 to 75 on version 1), so treat it as a signal to compare rounds, not a precise measurement. The problems you watched people hit matter more than the number.

Step 7: Report what you found

A usability report is short and goes to the people who'll fix things:

• The tasks, and how many people completed each one.
• Each problem: what happened, how many users hit it, its severity, and a screenshot or short clip.
• What worked well, so nobody "fixes" it.
• Recommendations, most severe first.

Keep it to problems you saw. "Three of five users typed their phone with +91 and lost the whole form" moves a team. "Users may find validation unclear" doesn't.

Common mistakes

• Testing with colleagues. They know the product and the words. Recruit people like your real users.
• Helping during the session. The moment you point, you've lost the finding.
• Asking "do you like it?" People are polite. Watch what they do instead.
• Testing too late. A usability finding on a sketch costs an eraser. On a finished release it costs a sprint.
• One round of five, then done. Five users miss many rarer problems. Fix, then test again.
• Only reviewing, never watching. A heuristic review finds the obvious problems. Users find the rest.
• Testing on a laptop for an app people use on phones. Use the real device, the real network, the real situation.

What's next

In Part 11 of Testing in Practice we write your first Selenium test: Selenium WebDriver with Python and pytest, against the FoodNow checkout, and how it compares with Playwright.

ProvaLab's test management system keeps usability findings next to the rest of your testing: write each session task as a test case, record each session's result, and file what users hit as issues with screenshots attached and a severity, linked to the task that found them. The checks that can be automated, like the label and tap-target checks in Step 1, can run as ordinary pytest tests, and ProvaLab's pytest plugin reports them on every run. See it in the two-minute product tour.

UAT is the last test level before release. The people who'll use the software, or the people who represent them, try it on realistic tasks and decide whether to accept it. It's often done badly: users are handed a build on a Friday afternoon, click through a script, and sign. That's a rubber stamp, not a test.

This post shows how to plan and run UAT for a new FoodNow feature, order for later. It includes two things that really ran:

• A dry run on a UAT-like server, which found a bug that every system test had passed.
• A script that checks the UAT results against the exit criteria, which caught a gap in how a retest was recorded.

The UAT sessions themselves need real people, so the session results in Step 6 are an example, and we say so where they appear.

What UAT is, and what it isn't

UAT is not a second round of system testing by people who aren't testers. If users are finding crashes and wrong totals, the build wasn't ready for UAT. Fix the process before that, not the users.

The ISTQB names several kinds of acceptance testing. You'll hear these:

• User acceptance testing: the users check it fits their work. This post.
• Operational acceptance testing: the people who'll run it check backups, monitoring, restarts and the release steps.
• Contract and regulatory acceptance testing: checks against what a contract or a law requires.
• Alpha and beta testing: for products sold to many customers. Alpha is with users at your site, beta is with users in their own environment.

The feature: order for later

Office managers asked for a way to order lunch for a meeting in advance. The story:

As an office manager, I want to choose a delivery slot when I order, so that lunch arrives when the meeting starts, not when I happened to order it.

The acceptance criteria the team built to:

• Slots are 30 minutes long, on the hour and half hour.
• The first slot is at least 45 minutes from now (cooking and riding time).
• Slots go up to 3 hours ahead.
• Every slot is within kitchen hours, 11:00 to 23:00.

Here's the code that picks the slots, in foodnow/app.py. The page that shows them is a small FastAPI app with an HTML form:

from datetime import datetime, timedelta

OPENS, CLOSES = 11, 23          # kitchen hours
LEAD = timedelta(minutes=45)    # cooking + riding time
AHEAD = timedelta(hours=3)


def now() -> datetime:
    return datetime.now()


def slots(at: datetime) -> list[datetime]:
    first = (at + LEAD).replace(second=0, microsecond=0)
    first += timedelta(minutes=(30 - first.minute % 30) % 30)
    out, t = [], first
    while t <= at + AHEAD:
        day = t.replace(hour=0, minute=0)
        if day + timedelta(hours=OPENS) <= t <= day + timedelta(hours=CLOSES, minutes=-30):
            out.append(t)
        t += timedelta(minutes=30)
    return out

And the team's system tests, which pass:

def test_slots_start_after_the_lead_time_and_are_half_hours():
    at = datetime(2026, 11, 11, 19, 10)
    s = fn.slots(at)
    assert s[0] == datetime(2026, 11, 11, 20, 0)
    assert all(b - a == timedelta(minutes=30) for a, b in zip(s, s[1:]))
    assert s[-1] <= at + fn.AHEAD


def test_no_slots_after_closing():
    assert fn.slots(datetime(2026, 11, 11, 22, 30)) == []


def test_page_offers_slots_from_now():
    html = client.get("/").text
    first = fn.slots(fn.now())[0]
    assert f"{first:%H:%M} –" in html


def test_scheduling_confirms_the_slot():
    r = client.post("/schedule", data={"slot": "20:00"})
    assert "between 20:00 and 20:30" in r.text

$ pytest -q
....                                                                     [100%]
4 passed, 1 deselected, 1 warning in 0.46s

$ TZ=UTC pytest -q
....                                                                     [100%]
4 passed, 1 deselected, 1 warning in 0.40s

They pass with the machine's clock in India time and in UTC. (The deselected test is the one we add in Step 4. The warning is a deprecation notice from Starlette's test client.) Keep datetime.now() in mind.

A correction, found later: test_page_offers_slots_from_now also depends on the real clock in a second way. Between 21:46 and 07:59 there are no slots left, and the test fails with IndexError. We ran these tests in the evening, so we missed it. Part 14 shows how it turned up in a nightly-style run, and the fix: give the test a fixed time.

Step 1: Plan, and agree the exit criteria first

A UAT plan is short. It answers:

• Who tests? People who'll really use it. For order for later: two office managers or team assistants who order for meetings, and one night-shift lead, because they order late. Not developers, and not the product owner alone.
• What do they do? Three to six realistic tasks (Step 3).
• Where and when? The UAT environment, on their own phones, in a 45-minute session each, in the week before release.
• Who decides? One named person who signs the acceptance. Here, the head of FoodNow for Business.
• What counts as accepted? The exit criteria.

Agree the exit criteria before UAT starts. Afterwards, there's pressure to call whatever happened "good enough". Ours:

1. Every must-have task is completed by at least two testers.
2. No open critical or major defects.
3. Minor defects are either fixed or accepted in writing by the product owner.
4. Change requests are logged, and don't block acceptance.

Step 2: Check the entry criteria

UAT wastes everyone's time if the build isn't ready. Before inviting anyone, check:

• The build has passed system testing, and the known defects are listed.
• The UAT environment is set up like production: same configuration, same kind of data, same integrations.
• Testers have accounts, and there's realistic data (menus, addresses, payment test cards).
• Testers know how to report a problem, and who to ask when they're stuck.

"Like production" is the one teams skip, and the next step shows why it matters.

Step 3: Write the scenarios as tasks, not steps

A system test script says exactly what to click. A UAT scenario gives the user a goal, and you watch how they reach it:

The script tells you whether the buttons work, which system testing already did. The task tells you whether a real user can find the feature, choose the right slot and trust the confirmation. If they hesitate, go back, or ask "is this the right time?", that's a finding.

Our scenarios:

• S1 (must have) Order lunch for a meeting that starts in about two hours.
• S2 (must have) Choose a slot, then change your mind before paying.
• S3 (must have) It's late evening. Order dinner for the night shift.
• S4 (nice to have) Order tomorrow's lunch today.
• S5 (nice to have) Read the confirmation aloud to a colleague: when does the food arrive?

S4 is there on purpose. The story only promised the next three hours. If users expect tomorrow, it's a change request, and that's worth knowing too.

Step 4: Dry run on the UAT environment

Before users arrive, a tester does the scenarios on the UAT environment, as a user would: on a phone, in the users' time zone. This catches the problems that would waste a session, and the ones the team's own setup hides.

Our UAT server runs in a container, and like most containers, its clock is set to UTC. The users are in India, 5 hours 30 minutes ahead. We started the server that way:

TZ=UTC uvicorn foodnow.app:app --port 8077

Then ran scenario S1 in a browser set up like the users' phones: a phone-sized screen, India's time zone and Indian English. This short Playwright script records what the tester saw:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    phone = browser.new_context(
        viewport={"width": 412, "height": 915}, device_scale_factor=2,
        is_mobile=True, has_touch=True,
        timezone_id="Asia/Kolkata", locale="en-IN",
    )
    page = phone.new_page()
    page.goto("http://localhost:8077/")
    print("time on the phone:", page.evaluate(
        "new Date().toLocaleTimeString('en-IN', {hour: '2-digit', minute: '2-digit', hour12: false})"))
    print("slots offered:   ", page.locator("label").all_inner_texts())
    page.locator("input[name=slot]").first.check()
    page.get_by_role("button", name="Schedule delivery").click()
    print("confirmation:    ", page.locator("#confirm").inner_text())
    browser.close()

time on the phone: 19:55
slots offered:     ['15:30 – 16:00', '16:00 – 16:30', '16:30 – 17:00', '17:00 – 17:30']
confirmation:      Your order will arrive between 15:30 and 16:00 today.

At 19:55, the app offered a delivery at 15:30, four and a half hours ago, and confirmed it.

(We drew the phone frame and its clock around the real screenshots. The time is the one the browser reported in the run.)

Why did every system test pass? datetime.now() returns the server's local time. On the developers' laptops, that's India time, so everything looked right. The test that checks the page, test_page_offers_slots_from_now, compares the page with slots(fn.now()), the same wrong clock, so it passed on the UTC machine too. A test that computes the expected value the same way as the code shares its mistakes. The other tests used fixed times, so they never asked "what time is it for the customer?"

The fix: slots are in the kitchen's local time, whatever the server's clock says:

 from datetime import datetime, timedelta
+from zoneinfo import ZoneInfo
...
+KITCHEN_TZ = ZoneInfo("Asia/Kolkata")  # slots are in the kitchen's local time
...
 def now() -> datetime:
-    return datetime.now()
+    return datetime.now(KITCHEN_TZ).replace(tzinfo=None)

And a system test that would have caught it, so it can't come back:

def test_now_is_kitchen_time_whatever_the_server_clock(monkeypatch):
    monkeypatch.setenv("TZ", "UTC")  # like the UAT server
    time.tzset()
    try:
        kitchen = datetime.now(ZoneInfo("Asia/Kolkata")).replace(tzinfo=None)
        assert abs(fn.now() - kitchen) < timedelta(seconds=5)
    finally:
        monkeypatch.undo()
        time.tzset()

On the old code, it fails by 5 hours 30 minutes:

E       assert datetime.timedelta(seconds=19799, microseconds=999987) < datetime.timedelta(seconds=5)
1 failed, 4 passed, 1 warning in 0.48s

With the fix, 5 passed. The dry run again, on the same UTC server:

time on the phone: 19:55
slots offered:     ['21:00 – 21:30', '21:30 – 22:00', '22:00 – 22:30', '22:30 – 23:00']
confirmation:      Your order will arrive between 21:00 and 21:30 today.

19:55 plus 45 minutes is 20:40, so 21:00 is the first slot, and the last ends when the kitchen closes at 23:00. Had we skipped the dry run, the first tester would have hit this in the first minute, and the session would have been spent on a bug instead of on whether the feature fits their work.

Step 5: Run the sessions

What makes a UAT session useful:

• Give the task, then be quiet. Read the task out, and let the user work. Don't point. If you help, you're testing yourself.
• Ask them to think aloud. "I'm looking for... I expected this to say..." is where the findings are.
• Write down what they do, not just what they say. Hesitations, wrong taps, going back. Users often blame themselves for the software's problems: "Sorry, I'm not good with apps."
• Use their devices, at their pace. Not a fast office laptop with the developer's account.
• Record each result: the scenario, the tester, pass or fail, and what they found.

Step 6: Triage, then check the exit criteria

The results below are an example, written for this post to show the process. They are not from real users. Picture this at the end of the sessions:

scenario,must_have,tester,result,finding,type,severity,status
S1 Schedule lunch for a meeting,yes,Office manager,pass,,,,
S1 Schedule lunch for a meeting,yes,Team assistant,pass,,,,
S2 Change your mind before paying,yes,Office manager,pass,,,,
S2 Change your mind before paying,yes,Team assistant,fail,Back button loses the chosen slot,defect,major,fixed and retested
S3 Late evening order,yes,Night-shift lead,pass,,,,
S3 Late evening order,yes,Team assistant,pass,,,,
S4 Order for tomorrow,no,Office manager,fail,Wants to book tomorrow's lunch today,change request,,backlog
S5 Read the confirmation aloud,no,Team assistant,pass,"""Today"" was unclear just after midnight",defect,minor,accepted by product owner

Every finding is triaged into one of three kinds:

• Defect: the software doesn't do what was agreed. The back button losing the slot breaks S2. Major, fixed, retested.
• Change request: it does what was agreed, but users want something else. Booking for tomorrow was never in the story. It goes to the backlog, and it doesn't block acceptance. It's still valuable: the next story is written.
• Wording, training or documentation: the software is right, but users misread it. "Today" just after midnight is confusing. Minor, and the product owner accepted it for now.

Arguing about which kind a finding is happens in every UAT. Decide with the acceptance criteria from the story, not with how strongly someone feels.

This script checks the results against the exit criteria from Step 1. It's go_no_go.py:

"""Check UAT results against the exit criteria agreed before UAT started."""
import csv
import sys
from collections import defaultdict

MIN_TESTERS = 2  # each must-have scenario passed by at least two people
BLOCKING = {"critical", "major"}
CLOSED = {"fixed and retested", "accepted by product owner"}

rows = list(csv.DictReader(open(sys.argv[1], encoding="utf-8")))
problems = []

passes = defaultdict(set)
for r in rows:
    if r["must_have"] == "yes" and r["result"] == "pass":
        passes[r["scenario"]].add(r["tester"])
for s in sorted({r["scenario"] for r in rows if r["must_have"] == "yes"}):
    if len(passes[s]) < MIN_TESTERS:
        problems.append(f"{s}: passed by {len(passes[s])} tester(s), need {MIN_TESTERS}")

for r in rows:
    if r["type"] == "defect" and r["status"] not in CLOSED:
        if r["severity"] in BLOCKING:
            problems.append(f"open {r['severity']} defect: {r['finding']}")
    if r["type"] == "defect" and r["severity"] in BLOCKING and r["status"] == "accepted by product owner":
        problems.append(f"{r['severity']} defect accepted instead of fixed: {r['finding']}")

defects = [r for r in rows if r["type"] == "defect"]
changes = [r for r in rows if r["type"] == "change request"]
print(f"{len(rows)} results, {len(defects)} defects, {len(changes)} change requests")
for r in changes:
    print(f"  change request (not a blocker): {r['finding']} -> {r['status']}")
if problems:
    print("NO GO")
    for p in problems:
        print("  -", p)
    sys.exit(1)
print("GO: exit criteria met, ready for sign-off")

$ python go_no_go.py uat-results.csv
8 results, 2 defects, 1 change requests
  change request (not a blocker): Wants to book tomorrow's lunch today -> backlog
NO GO
  - S2 Change your mind before paying: passed by 1 tester(s), need 2

We expected a GO: the defect's status says "fixed and retested". But the retest was never recorded as a result. The sheet only shows the failure. As far as the evidence goes, only one person has completed S2. That's a real gap that happens all the time: the fix is mentioned in a chat, and the record still says it failed. Recording the retest:

S2 Change your mind before paying,yes,Team assistant,pass,Retest after the back-button fix,,,

$ python go_no_go.py uat-results.csv
9 results, 2 defects, 1 change requests
  change request (not a blocker): Wants to book tomorrow's lunch today -> backlog
GO: exit criteria met, ready for sign-off

And for comparison, the results as they stood on the first day, with the back-button defect still open:

NO GO
  - S2 Change your mind before paying: passed by 1 tester(s), need 2
  - open major defect: Back button loses the chosen slot

A script doesn't make the decision. It makes sure the decision is made against the criteria everyone agreed, with the evidence in front of them, and not from memory at the end of a long week.

Step 7: Sign-off

The sign-off is a short record, signed by the named person from Step 1:

• What was tested: the build version and the environment.
• Results: the scenarios, testers and outcomes, and the exit criteria result.
• Known issues accepted: the minor "today" wording, with who accepted it.
• Change requests raised: booking for tomorrow, now in the backlog.
• Decision: accepted, accepted with conditions, or not accepted.

If the decision is "not accepted", that's UAT working, not failing. It's much cheaper than finding out from customers.

Common mistakes

• Treating UAT as system testing by other people. If users find crashes and wrong totals, the build wasn't ready. Use entry criteria.
• Step-by-step scripts. They tell you the buttons work. Tasks tell you whether people can do their job.
• A UAT environment that isn't like production. Ours had a UTC clock, and the bug only existed there. Do a dry run on it, as a user, before inviting anyone.
• Helping the user. The moment you point at the right button, you've lost the finding.
• Deciding the exit criteria afterwards. Agree them in the plan, then check the results against them.
• Treating every finding as a bug. Change requests are useful, but they're not defects. Log them, and don't let them block a release that does what was agreed.
• Fixes that are never retested on the record. Step 6 shows how a "fixed and retested" status can hide a missing result.

What's next

In Part 10 of Testing in Practice we look at usability testing: how to watch five people use your app and find most of the problems that make users give up, and how it differs from UAT.

ProvaLab's test management system keeps UAT in the same place as the rest of your testing. Write the UAT scenarios as test cases, run them as a test run in the UAT build's version, and record the results, with defects linked to the case that found them. When it's time to decide, the run shows what passed, what failed and what was retested, next to the system tests for the same release. See it in the two-minute product tour.

Most of the web still lets them down. The 2026 WebAIM Million scan of the top million home pages found detectable WCAG failures on 95.9% of them. The most common problem, for the eighth year running, was low-contrast text.

This post starts the Testing in Practice series. It covers what accessibility testing checks, then a hands-on run: we plant seven accessibility bugs in a small checkout page and see which ones an automated scan finds, which ones our own tests find, and which one only a person notices. Every command and every line of output below was run while writing this post.

What accessibility testing checks

Accessibility testing (often shortened to a11y: an "a", 11 letters, a "y") checks that people with disabilities can perceive, understand and use your app. The yardstick almost everyone uses is WCAG, the W3C's Web Content Accessibility Guidelines. The current version is WCAG 2.2, published in October 2023.

WCAG is built on four principles, easy to remember as POUR:

Each guideline has success criteria at three levels: A (the minimum), AA (the usual target) and AAA (the strictest). When a contract, a law or a client says "accessible", it almost always means WCAG 2.1 or 2.2 level AA. In the EU, the European Accessibility Act has applied since 28 June 2025 to many products and services, online shops included. So this isn't just good practice: for many teams, it's a legal requirement.

Three layers of accessibility testing

No single check finds everything. Good teams use three layers:

1. An automated scan (axe-core, Lighthouse, WAVE). It checks the page against rules in about a second: contrast, missing alt text, missing labels, missing page language. Cheap enough to run on every build.
2. Scripted checks you write yourself: can a keyboard reach and press every control, and does every control have a name a screen reader can read? These are ordinary automated tests.
3. A person: tabbing through the page, zooming to 200%, listening with a screen reader, judging whether a label actually makes sense. No tool can tell you that "Click here" is a bad link text for a blind user scanning a list of links.

How much does the first layer catch? Deque, the company behind axe-core, reported that automated tests found 57% of issues across more than 2,000 of its audits. That's by volume of issues, and it's the vendor's own figure. Read it as "a lot, but nowhere near all", which is exactly what we'll see below.

The page under test

Our page is a checkout from the food delivery app that this blog uses in every example. It has a delivery tip box (0 to 200, as in How to write test cases), a button to remove the tip and a Pay button. It looks fine. We've planted seven accessibility bugs in it:

<!doctype html>
<html>
<head>
  <meta charset="utf-8">
  <title>Checkout · FoodNow</title>
  <style>
    body { font-family: system-ui, sans-serif; max-width: 420px; margin: 40px auto; color: #111; }
    .note { color: #b5b5b5; font-size: 14px; }
    .pay { background: #16a34a; color: #fff; padding: 12px; text-align: center; border-radius: 8px; cursor: pointer; }
    .remove { border: 0; background: none; font-size: 18px; cursor: pointer; }
  </style>
</head>
<body>
  <img src="logo.png" width="120" height="40">
  <h1>Your order</h1>
  <p>Margherita pizza · 500</p>

  <input id="tip" type="number" placeholder="Delivery tip (0 to 200)">
  <button class="remove" onclick="document.getElementById('tip').value=''">×</button>
  <p class="note">The tip goes to your rider in full.</p>

  <div class="pay" onclick="pay()">Pay 500</div>
  <p id="status"></p>

  <script>
    function pay() { document.getElementById("status").textContent = "Order placed"; }
  </script>
</body>
</html>

Can you spot all seven before reading on? Most people find two or three.

Step 1: Set up (2 minutes)

You need Python and uv. We used these versions:

axe-core is a JavaScript library. We inject it into the page ourselves, so you can see exactly what runs. Download its single file once:

npm pack axe-core@4.14.0
tar xzf axe-core-4.14.0.tgz package/axe.min.js --strip-components=1

The project is four files: the page in site/, the axe file, a conftest.py that serves the page, and the tests.

a11y/
├── axe.min.js
├── conftest.py
├── site/
│   ├── checkout.html
│   └── logo.png
└── test_accessibility.py

conftest.py starts a small web server on a free port for the test session, so the tests open a real http:// page:

import functools, http.server, pathlib, threading
import pytest

SITE = pathlib.Path(__file__).parent / "site"


class QuietHandler(http.server.SimpleHTTPRequestHandler):
    def log_message(self, *args):  # keep the test output readable
        pass


@pytest.fixture(scope="session")
def base_url():
    """Serve site/ on a free local port for the whole test session."""
    handler = functools.partial(QuietHandler, directory=SITE)
    server = http.server.ThreadingHTTPServer(("127.0.0.1", 0), handler)
    threading.Thread(target=server.serve_forever, daemon=True).start()
    yield f"http://127.0.0.1:{server.server_port}"
    server.shutdown()

If you haven't used Playwright before, Your first Playwright test covers installing it and its browsers.

Step 2: Write three accessibility tests

One test per layer we can automate:

import pathlib
from playwright.sync_api import Page, expect

AXE = pathlib.Path(__file__).parent / "axe.min.js"
WCAG_AA = ["wcag2a", "wcag2aa", "wcag21a", "wcag21aa", "wcag22aa"]


def axe_violations(page: Page) -> list[dict]:
    """Run axe-core on the page and return the WCAG A/AA rules it breaks."""
    page.add_script_tag(path=AXE)
    result = page.evaluate(
        "tags => axe.run(document, { runOnly: { type: 'tag', values: tags } })", WCAG_AA
    )
    return result["violations"]


def describe(violations: list[dict]) -> str:
    lines = []
    for v in violations:
        lines.append(f"{v['id']} ({v['impact']}): {v['help']}")
        for node in v["nodes"]:
            lines.append(f"    {node['target'][0]}")
    return "\n".join(lines)


def test_checkout_has_no_wcag_violations(page: Page, base_url: str):
    page.goto(f"{base_url}/checkout.html")
    violations = axe_violations(page)
    assert not violations, "\n" + describe(violations)


def test_checkout_works_with_the_keyboard_alone(page: Page, base_url: str):
    page.goto(f"{base_url}/checkout.html")
    page.get_by_role("spinbutton", name="Delivery tip").fill("50")
    # From the tip box, Tab until the Pay button has focus (no mouse).
    for _ in range(5):
        page.keyboard.press("Tab")
        if page.evaluate("document.activeElement.textContent.trim()").startswith("Pay"):
            break
    page.keyboard.press("Enter")
    expect(page.locator("#status")).to_have_text("Order placed")


def test_controls_have_names_a_screen_reader_can_read(page: Page, base_url: str):
    page.goto(f"{base_url}/checkout.html")
    expect(page.locator("body")).to_match_aria_snapshot(
        """
        - spinbutton "Delivery tip (0 to 200)"
        - button "Remove tip"
        - button "Pay 500"
        """
    )

What each one does:

• The axe scan limits itself to the WCAG A and AA rules (the wcag2a … wcag22aa tags), so it checks the level you're actually held to. describe() turns axe's large JSON answer into one readable line per problem.
• The keyboard test never clicks. It types the tip, presses Tab until Pay has focus, and presses Enter. That's how a keyboard or switch user pays.
• The names test checks the accessibility tree: the version of the page that a screen reader reads out. Playwright's to_match_aria_snapshot compares the role and name of each control with what we expect. A button must be announced as a button, with a name that says what it does.

Step 3: Run it and read the failures

uv run --no-project --python 3.12 --with pytest==9.2.0 --with pytest-playwright==0.10.0 --with playwright==1.64.0 python -m pytest -q

All three fail. Here's the important part of each.

The axe scan found four problems under three rules:

E       AssertionError:
E         color-contrast (serious): Elements must meet minimum color contrast ratio thresholds
E             .note
E             div
E         html-has-lang (serious): <html> element must have a lang attribute
E             html
E         image-alt (critical): Images must have alternative text
E             img

axe also explains each one. For the contrast failures it gave the exact numbers:

.note  Element has insufficient color contrast of 2.05 (foreground color: #b5b5b5, background color: #ffffff, font size: 10.5pt (14px), font weight: normal). Expected contrast ratio of 4.5:1
div    Element has insufficient color contrast of 3.29 (foreground color: #ffffff, background color: #16a34a, font size: 12.0pt (16px), font weight: normal). Expected contrast ratio of 4.5:1

The grey note was an obvious one. The white-on-green Pay button is a surprise to most people: it looks perfectly readable, and it's a very common brand green, but at 16px it's well short of the 4.5:1 that WCAG AA asks for normal-size text.

The keyboard test typed the tip, pressed Tab, pressed Enter, and nothing happened:

E       AssertionError: Locator expected to have text 'Order placed'
E       Actual value:
...
E       Aria snapshot:
E       - img
E       - heading "Your order" [level=1]
E       - paragraph: Margherita pizza · 500
E       - spinbutton "Delivery tip (0 to 200)": "50"
E       - button "×"
E       - paragraph: The tip goes to your rider in full.
E       - text: Pay 500
E       - paragraph

Playwright helpfully printed the accessibility tree, and it shows why: Pay 500 is just text, not a button. A <div> with an onclick works with a mouse, but the keyboard skips it and a screen reader doesn't announce it as something you can press. A keyboard user simply cannot pay. axe didn't report this, because a <div> with a click handler is invisible to its rules: nothing in the HTML says it's meant to be a button.

The names test failed on two lines:

E       AssertionError: Locator expected to match Aria snapshot '
E               - spinbutton "Delivery tip (0 to 200)"
E               - button "Remove tip"
E               - button "Pay 500"
E               '
...
E       - spinbutton "Delivery tip (0 to 200)"
E       - button "×"
E       - paragraph: The tip goes to your rider in full.
E       - text: Pay 500

The remove button's name is "×". axe was satisfied, because the button does have a name. But a screen reader reads that character out as "times" or "multiplication sign", which tells a blind user nothing. And Pay is missing as a button, again.

The run took 11 seconds instead of one because both Playwright assertions wait up to five seconds for the page to become right before giving up.

Step 4: The one only a person finds

The tests found six of the seven problems. The seventh: the tip box has no label, only a placeholder. The placeholder gives the box an accessible name, so axe and our names test both passed it. But the moment you start typing, the placeholder disappears. Someone with a memory or attention difficulty who comes back to the field no longer knows what it was for. And placeholder grey text is usually low-contrast too. Only a person looking at how the form behaves catches that.

That's the shape of accessibility testing in one picture. The scan is fast, so run it on every build. It found four of the seven in about a second. Two short tests of your own found two more. A person found the last one, and a person is also the only one who can tell you whether the page is actually pleasant to use with a screen reader, which no test checks.

Step 5: Fix them and run again

Every fix is small:

-<html>
+<html lang="en">
-    .note { color: #b5b5b5; font-size: 14px; }
-    .pay { background: #16a34a; color: #fff; padding: 12px; text-align: center; border-radius: 8px; cursor: pointer; }
+    .note { color: #6b7280; font-size: 14px; }
+    .pay { background: #15803d; color: #fff; padding: 12px; width: 100%; border: 0; border-radius: 8px; font: inherit; cursor: pointer; }
-  <img src="logo.png" width="120" height="40">
+  <img src="logo.png" width="120" height="40" alt="FoodNow">
-  <input id="tip" type="number" placeholder="Delivery tip (0 to 200)">
-  <button class="remove" onclick="document.getElementById('tip').value=''">×</button>
+  <label for="tip">Delivery tip (0 to 200)</label>
+  <input id="tip" type="number" min="0" max="200">
+  <button class="remove" aria-label="Remove tip" onclick="document.getElementById('tip').value=''">×</button>
-  <div class="pay" onclick="pay()">Pay 500</div>
-  <p id="status"></p>
+  <button class="pay" onclick="pay()">Pay 500</button>
+  <p id="status" role="status"></p>

• lang="en" tells a screen reader which language and pronunciation to use.
• Darker colours: a slightly darker grey and a slightly darker green both pass 4.5:1, and the design looks almost the same.
• alt="FoodNow": the logo now has a text alternative. A purely decorative image would get alt="" instead, so screen readers skip it.
• A real <label> that stays visible while you type.
• aria-label="Remove tip": the button still shows ×, but is announced as "Remove tip".
• A real <button> for Pay. This one change makes it reachable with Tab, pressable with Enter and Space, and announced as a button. Use the native element and you get all of that for free.
• role="status" on the message, so a screen reader announces "Order placed" without the user having to go looking for it.

Run the same command again:

...                                                                      [100%]
3 passed in 1.24s

A 10-minute manual check for every new page

The tests now guard against these bugs coming back. For anything new, spend ten minutes as a person:

1. Keyboard only. Unplug the mouse. Tab through the whole page. Can you see where focus is at all times? Can you reach, use and leave every control, including menus and pop-ups?
2. Zoom to 200%, then 400%. Does text reflow into one column, or does it overlap, cut off, or make you scroll sideways?
3. Screen reader for two minutes: NVDA (free) on Windows, VoiceOver on Mac and iPhone, TalkBack on Android. Listen to the form. Does every field say what it is? Are errors read out?
4. Headings. Do they form a sensible outline (one h1, then h2s)? Screen-reader users jump between headings to skim.
5. Errors. Submit the form wrong. Is the error in text, not only red colour, and does it say how to fix it?
6. Moving things. Anything that moves, blinks or scrolls by itself for more than five seconds needs a way to pause it, and nothing should flash more than three times a second.

Common mistakes

• Trusting a green scan. "axe found nothing" means "axe found nothing it can detect". As we just saw, it can miss the bug that stops a keyboard user from paying.
• Adding ARIA instead of using native elements. <div role="button" tabindex="0"> plus key handlers is more code and more ways to get it wrong than <button>. The first rule of ARIA is: don't use ARIA if a native element does the job.
• Testing once, at the end. Accessibility fixes are cheap in a component and expensive across 200 finished pages. Put the scan in CI from the first page.
• Fixing the tool's list and calling it done. Users with disabilities are the real test. If you can, include them in usability sessions.

Put it in your pipeline

The scan test is an ordinary pytest test, so it runs wherever your other tests run. In a real suite, scan every important page and every important state: the menu open, the form showing errors, the modal on screen, because axe only sees the page as it is at that moment. Many teams fail the build on serious and critical problems, and report moderate and minor ones without failing.

What's next

In Part 2 of Testing in Practice we test an API with Postman, then run the same collection from the command line with Newman, so it can run in CI.

ProvaLab's QA agent can do the automated part for you: ask it to check a page's accessibility and it writes an axe-core test like the one above, runs it in a real browser, and reports the WCAG A and AA problems by impact. The keyboard check and the person with a screen reader are still yours. See the agent in the two-minute product tour.

This post turns Postman requests into real tests. We build a small collection for a food delivery API, add checks to every request, pass a login token from one request to the next, and run the whole thing from the command line with Newman, Postman's open-source runner. Then we feed it a CSV of boundary values, and it finds a bug that the "normal" tests walked straight past.

Every command and every line of output below was run while writing this post.

If APIs are new to you, read API testing explained first. It covers methods, status codes and what to check in a response.

What you'll build

• An API to test: a tiny FoodNow orders API, running on your own machine.
• A Postman collection: 7 requests in two folders, with 13 checks.
• An environment: the base URL and login, kept out of the requests so you can point the same tests at local, staging or production.
• A data-driven run: one request, six tip values from a CSV file.
• A run you can put in CI: one command, an exit code, and a JUnit report.

We used these versions:

Step 1: Start the API under test (2 minutes)

The API has four endpoints: log in, read the menu, place an order, read an order. Orders take a delivery tip, and the requirement is the one from How to write test cases: a tip from 0 to 200.

Save this as api.py:

"""FoodNow orders API: just enough to test with Postman and Newman."""
import itertools
from fastapi import Depends, FastAPI, Header, HTTPException
from pydantic import BaseModel, Field

app = FastAPI(title="FoodNow")

USERS = {"asha@example.com": "pizza123"}
TOKENS: dict[str, str] = {}
MENU = {1: {"id": 1, "name": "Margherita pizza", "price": 500},
        2: {"id": 2, "name": "Paneer wrap", "price": 250}}
ORDERS: dict[int, dict] = {}
next_id = itertools.count(1001)


class Login(BaseModel):
    email: str
    password: str


class NewOrder(BaseModel):
    item_id: int
    tip: int = Field(0, ge=0, lt=200)  # the requirement says 0 to 200


def user(authorization: str = Header(None)) -> str:
    token = (authorization or "").removeprefix("Bearer ")
    if token not in TOKENS:
        raise HTTPException(401, "Log in first")
    return TOKENS[token]


@app.post("/login")
def login(body: Login):
    if USERS.get(body.email) != body.password:
        raise HTTPException(401, "Wrong email or password")
    token = f"tok-{len(TOKENS) + 1}"
    TOKENS[token] = body.email
    return {"token": token}


@app.get("/menu")
def menu():
    return list(MENU.values())


@app.post("/orders", status_code=201)
def create_order(body: NewOrder, email: str = Depends(user)):
    item = MENU.get(body.item_id)
    if not item:
        raise HTTPException(404, "No such item")
    order = {"id": next(next_id), "item": item["name"], "subtotal": item["price"],
             "tip": body.tip, "total": item["price"] + body.tip, "customer": email}
    ORDERS[order["id"]] = order
    return order


@app.get("/orders/{order_id}")
def get_order(order_id: int, email: str = Depends(user)):
    order = ORDERS.get(order_id)
    if not order or order["customer"] != email:
        raise HTTPException(404, "No such order")
    return order

There's a bug in it, the kind that ships every week. Look at the tip line if you want a head start. Otherwise, let's see whether our tests find it.

Start it with uv:

uv run --with fastapi==0.162.2 --with uvicorn==0.54.0 uvicorn api:app --port 8077

Step 2: The pieces of a Postman collection

A collection is a saved set of requests, in folders, with everything Postman needs to send them again. Four features turn it from a bookmark list into a test suite:

In the app, you'll find these on each request's Scripts tab (Pre-request and Post-response), and the variables on the collection's Variables tab. Postman saves the whole collection as a JSON file, and that file is what Newman runs. It also goes into git next to your code, so a pull request can show what changed in the tests.

We wrote our collection as that JSON file directly. To see it in the app, use Import and choose the file: you get the same folders, requests and scripts.

Step 3: Write the requests and their tests

Our collection has two folders.

Happy path, in this order, because each request uses what the one before it saved:

1. Log in: POST {{baseUrl}}/login with the email and password from the environment. The test saves the token:

   pm.test("status is 200", () => pm.response.to.have.status(200));
   const token = pm.response.json().token;
   pm.test("returns a token", () => pm.expect(token).to.be.a("string").and.not.empty);
   pm.collectionVariables.set("token", token);
   

2. Get the menu: GET {{baseUrl}}/menu. Checks the data, not just the status, and puts a ceiling on the response time:

   pm.test("status is 200", () => pm.response.to.have.status(200));
   const menu = pm.response.json();
   pm.test("pizza costs 500", () => {
       const pizza = menu.find(i => i.name === "Margherita pizza");
       pm.expect(pizza.price).to.eql(500);
   });
   pm.test("answers in under 500 ms", () => pm.expect(pm.response.responseTime).to.be.below(500));
   

3. Order with a tip of 50: POST {{baseUrl}}/orders with {"item_id": 1, "tip": 50}. It uses the collection's auth setting, Bearer token {{token}}, so every request sends the token from step 1 without you copying it anywhere. The test checks the arithmetic and saves the new order's id:

   pm.test("status is 201", () => pm.response.to.have.status(201));
   const o = pm.response.json();
   pm.test("total is subtotal + tip", () => pm.expect(o.total).to.eql(o.subtotal + 50));
   pm.collectionVariables.set("orderId", o.id);
   

4. Get that order: GET {{baseUrl}}/orders/{{orderId}}, and check it's the same order:

   pm.test("status is 200", () => pm.response.to.have.status(200));
   pm.test("it is the order we just placed", () => {
       const o = pm.response.json();
       pm.expect(o.id).to.eql(Number(pm.collectionVariables.get("orderId")));
       pm.expect(o.tip).to.eql(50);
   });
   

Passing values from one request to the next like this is called chaining. It's what makes a collection test a whole flow (log in, order, check the order) instead of single requests in isolation.

Errors, the negative tests:

5. Order without logging in: the same order with auth set to No Auth. Expect 401.

6. Order with a tip of 250: expect 422, and check the error points at the right field:

   pm.test("status is 422", () => pm.response.to.have.status(422));
   pm.test("the error names the tip field", () =>
       pm.expect(pm.response.json().detail[0].loc).to.include("tip"));
   

7. Order an item that does not exist: item_id 99. Expect 404.

The environment file, local.postman_environment.json, holds the three values that change from one place to another:

{
  "name": "Local",
  "values": [
    { "key": "baseUrl", "value": "http://localhost:8077", "enabled": true },
    { "key": "email", "value": "asha@example.com", "enabled": true },
    { "key": "password", "value": "pizza123", "enabled": true, "type": "secret" }
  ]
}

For staging you'd have a second file with a different baseUrl and a test account. The requests and tests don't change at all.

Step 4: Run it with Newman

Newman runs a collection from the terminal, with no app and no account. With Node.js installed:

npx newman@6.2.3 run foodnow.postman_collection.json -e local.postman_environment.json

FoodNow API

❏ Happy path
↳ Log in
  POST http://localhost:8077/login [200 OK, 142B, 23ms]
  ✓  status is 200
  ✓  returns a token

↳ Get the menu
  GET http://localhost:8077/menu [200 OK, 215B, 4ms]
  ✓  status is 200
  ✓  pizza costs 500
  ✓  answers in under 500 ms

↳ Order with a tip of 50
  POST http://localhost:8077/orders [201 Created, 234B, 3ms]
  ✓  status is 201
  ✓  total is subtotal + tip

↳ Get that order
  GET http://localhost:8077/orders/1001 [200 OK, 229B, 3ms]
  ✓  status is 200
  ✓  it is the order we just placed

❏ Errors
↳ Order without logging in
  POST http://localhost:8077/orders [401 Unauthorized, 160B, 3ms]
  ✓  status is 401

↳ Order with a tip of 250
  POST http://localhost:8077/orders [422 Unprocessable Entity, 265B, 3ms]
  ✓  status is 422
  ✓  the error names the tip field

↳ Order an item that does not exist
  POST http://localhost:8077/orders [404 Not Found, 157B, 3ms]
  ✓  status is 404

┌─────────────────────────┬─────────────────┬─────────────────┐
│                         │        executed │          failed │
├─────────────────────────┼─────────────────┼─────────────────┤
│              iterations │               1 │               0 │
├─────────────────────────┼─────────────────┼─────────────────┤
│                requests │               7 │               0 │
├─────────────────────────┼─────────────────┼─────────────────┤
│            test-scripts │               7 │               0 │
├─────────────────────────┼─────────────────┼─────────────────┤
│      prerequest-scripts │               0 │               0 │
├─────────────────────────┼─────────────────┼─────────────────┤
│              assertions │              13 │               0 │
├─────────────────────────┴─────────────────┴─────────────────┤
│ total run duration: 160ms                                   │
├─────────────────────────────────────────────────────────────┤
│ total data received: 484B (approx)                          │
├─────────────────────────────────────────────────────────────┤
│ average response time: 6ms [min: 3ms, max: 23ms, s.d.: 6ms] │
└─────────────────────────────────────────────────────────────┘

Thirteen checks, all green, in 160 milliseconds. You can see the chaining working: the order request got 201, and the next request fetched /orders/1001, the id it saved.

And the API still has a bug in it. Every one of these tests is correct. They just never asked the question that matters.

Step 5: Drive it from a CSV of boundary values

In Test design techniques we said bugs love edges: test at and around each boundary. For a tip from 0 to 200 that's six values: −1, 0, 1, 199, 200 and 201. Writing six requests would be copy-paste. Instead, write one request and give Newman a data file. It runs the collection once per row, an iteration, and the test reads that row's values.

tips.csv:

tip,expected_status
-1,422
0,201
1,201
199,201
200,201
201,422

A second, small collection has one request, Order with the tip from tips.csv, with the body {"item_id": 1, "tip": {{tip}}}. Here {{tip}} comes from the CSV row. Its pre-request script logs in on the first iteration only, using pm.sendRequest:

// Log in once, before the first request, and keep the token for the others.
if (!pm.collectionVariables.get("token")) {
    pm.sendRequest({
        url: pm.environment.get("baseUrl") + "/login",
        method: "POST",
        header: { "Content-Type": "application/json" },
        body: { mode: "raw", raw: JSON.stringify({
            email: pm.environment.get("email"),
            password: pm.environment.get("password"),
        }) },
    }, (err, res) => pm.collectionVariables.set("token", res.json().token));
}

And its test compares the status with the row's expected_status:

const tip = pm.iterationData.get("tip");
const expected = Number(pm.iterationData.get("expected_status"));
pm.test(`tip ${tip} → ${expected}`, () => pm.response.to.have.status(expected));

Run it with -d for the data file:

npx newman@6.2.3 run tip-boundaries.postman_collection.json -e local.postman_environment.json -d tips.csv

Iteration 1/6
→ Order with the tip from tips.csv
  POST http://localhost:8077/login [200 OK, 142B, 25ms]
  POST http://localhost:8077/orders [422 Unprocessable Entity, 284B, 4ms]
  ✓  tip -1 → 422
...
Iteration 4/6
→ Order with the tip from tips.csv
  POST http://localhost:8077/orders [201 Created, 235B, 4ms]
  ✓  tip 199 → 201

Iteration 5/6
→ Order with the tip from tips.csv
  POST http://localhost:8077/orders [422 Unprocessable Entity, 265B, 2ms]
  1. tip 200 → 201

Iteration 6/6
→ Order with the tip from tips.csv
  POST http://localhost:8077/orders [422 Unprocessable Entity, 265B, 3ms]
  ✓  tip 201 → 422

  #  failure         detail
 1.  AssertionError  tip 200 → 201
     iteration: 5    expected response to have status code 201 but got 422
                     at assertion:0 in test-script
                     inside "Order with the tip from tips.csv"

A tip of exactly 200 is refused. The requirement says 0 to 200, so 200 must be accepted. The cause is the line we pointed at: lt=200 means "less than 200". It should be le=200, "less than or equal". It's the classic off-by-one at a boundary. The happy-path test (tip 50) and the obvious negative test (tip 250) both passed, because neither of them is anywhere near the edge.

Notice the login request appears only in iteration 1. The pre-request script saw the token on later iterations and skipped it.

Step 6: Fix it, and make the run CI-ready

The fix is two letters:

-    tip: int = Field(0, ge=0, lt=200)  # the requirement says 0 to 200
+    tip: int = Field(0, ge=0, le=200)  # the requirement says 0 to 200

Restart the API and run again, this time adding a JUnit report, the XML format almost every CI system can read and display:

npx newman@6.2.3 run tip-boundaries.postman_collection.json -e local.postman_environment.json -d tips.csv \
  --reporters cli,junit --reporter-junit-export results/tips.xml

  ✓  tip -1 → 422
  ✓  tip 0 → 201
  ✓  tip 1 → 201
  ✓  tip 199 → 201
  ✓  tip 200 → 201
  ✓  tip 201 → 422
│              assertions │               6 │               0 │

<testsuites name="FoodNow tip boundaries" tests="6" time="0.019">
  <testsuite name="Order with the tip from tips.csv" tests="1" failures="0" errors="0" time="0.003">
    <testcase name="tip -1 → 422" time="0.003" classname="OrderWithTheTipFromTipsCsv"/>
    <testcase name="tip 0 → 201" time="0.003" classname="OrderWithTheTipFromTipsCsv"/>

The first lines of the report (we've left out the generated ids and timestamps) show one test case per CSV row. The main collection still passes all 13 checks too. Two things make this ready for a pipeline:

• The exit code. Newman exits with 1 when any test fails (our first boundary run did) and 0 when all pass. That's all a CI job needs to go red or green.
• The JUnit file. CI tools turn it into a list of passed and failed tests on the build page, so nobody has to scroll through a log.

Run it in your CI system and you have an API test stage. A full CI pipeline gets its own post later in this series.

Common mistakes

• Tests that only check the status. 200 OK with the wrong total is still a bug. Check the values that matter: the price, the total, the id you just created.
• Hard-coded URLs and tokens. http://localhost:8077 typed into 40 requests means 40 edits for staging. Put it in the environment once. Same for tokens: log in in the collection, never paste one in.
• Requests that depend on each other by accident. Chaining on purpose is fine: log in, order, read the order. But if test 12 only passes because test 7 happened to create some data, a reordered or filtered run breaks for no visible reason. Keep each folder able to run on its own.
• Secrets in the exported files. Collections and environments end up in git. Mark passwords as secret, use test accounts, and pass real secrets from the CI system with --env-var "password=...".
• Only happy-path values. As the run above shows, a correct-looking suite can miss a boundary bug completely. Test the edges.

Postman, Newman, or code?

Many teams use both: Postman to explore and to share examples, code for the regression suite that grows over years. Your first API tests builds the code version of what we did here.

What's next

In Part 3 of Testing in Practice we catch the bugs functional tests can't see: visual regression testing with Playwright screenshots, baselines, masks and thresholds.

ProvaLab's QA AI Agent runs API test cases on its own and records every request and response against its test case, and AppSec's API fuzzing sends thousands of malformed and unauthorised requests to find what a hand-written collection misses. See it in the two-minute product tour.

Visual regression testing closes that gap. It takes a screenshot of the page, compares it with an approved one from before, and fails when the pixels change. This post builds one with Playwright, then shows the three things that make or break it in real life: the first run, the content that changes every time, and how much difference counts as "different".

Every command and every line of output below was run while writing this post. Along the way we found that the default settings let a real colour change through. The numbers are in Step 5.

What visual regression testing catches

It doesn't replace functional tests. A screenshot of a checkout that charges the wrong amount can look perfect. You want both.

How it works

The loop is always the same, whatever the tool:

The baseline is the approved screenshot, stored next to the tests and committed to git. The comparison isn't "are the files identical". It's "how many pixels differ, by how much colour", with two dials you control. We'll turn both.

Step 1: Set up (3 minutes)

Playwright's screenshot assertion, toHaveScreenshot, lives in its JavaScript/TypeScript test runner. The Python package we used in earlier posts doesn't have it built in, so this post uses TypeScript. You don't need to know much of it: the tests are a few lines each.

mkdir visual && cd visual
npm init -y
npm install -D @playwright/test@1.64.0
npx playwright install chromium

We used Node.js 20.19.5 and Playwright 1.64.0 on Linux.

The page under test is the FoodNow checkout from the accessibility post, with a little more design. It uses a shared stylesheet, site/base.css, which every FoodNow page loads:

/* Shared styles for every FoodNow page. */
* { box-sizing: border-box; }
body { font-family: "DejaVu Sans", Verdana, sans-serif; margin: 0; background: #f1f5f9; color: #0f172a; }
button { font: inherit; cursor: pointer; }

The page shows the order, a tip box with a × button to remove the tip, the total, a line with the time of the order, and a Pay button. A short Node server, server.mjs, serves the site/ folder on port 4173. The config starts it before the tests:

// playwright.config.ts
import { defineConfig, devices } from "@playwright/test";

export default defineConfig({
  testDir: "./tests",
  use: { baseURL: "http://localhost:4173" },
  webServer: { command: "node server.mjs", url: "http://localhost:4173/checkout.html", reuseExistingServer: true },
  projects: [{ name: "chromium", use: { ...devices["Desktop Chrome"] } }],
});

Step 2: One functional test, one visual test

// tests/checkout.spec.ts
import { test, expect } from "@playwright/test";

test("removing the tip updates the total", async ({ page }) => {
  await page.goto("/checkout.html");
  await page.getByRole("button", { name: "Remove tip" }).click();
  await expect(page.locator("#total")).toHaveText("500");
  await expect(page.getByRole("button", { name: "Pay 500" })).toBeVisible();
});

test("checkout looks right", async ({ page }) => {
  await page.goto("/checkout.html");
  await expect(page).toHaveScreenshot("checkout.png");
});

The first test is the kind you already write. The second is the whole of a visual test: one line.

Step 3: The first run fails, on purpose

npx playwright test

  ✘  2 [chromium] › tests/checkout.spec.ts:10:5 › checkout looks right (341ms)

    Error: A snapshot doesn't exist at tests/checkout.spec.ts-snapshots/checkout-chromium-linux.png, writing actual.

  1 failed
  1 passed (1.3s)

That's expected. There's nothing to compare with yet, so Playwright saves this screenshot as the baseline and fails, to make sure a person looks at it before it's trusted. Open the PNG and check it's actually right. Everything after this is measured against it.

Note the file name: checkout-chromium-linux.png. The browser and the operating system are part of it, because fonts and anti-aliasing render slightly differently on Linux, macOS and Windows. More on that in the mistakes section.

Step 4: The second run fails too. Why?

Run it again, change nothing:

  ✓  1 [chromium] › tests/checkout.spec.ts:3:5 › removing the tip updates the total (149ms)
  ✘  2 [chromium] › tests/checkout.spec.ts:10:5 › checkout looks right (568ms)

    Error: expect(page).toHaveScreenshot(expected) failed

      15 pixels (ratio 0.01 of all image pixels) are different.

    Call log:
      - taking page screenshot
        - disabled all CSS animations
      - waiting for fonts to load...
      - fonts loaded
      - 15 pixels (ratio 0.01 of all image pixels) are different.
      - waiting 100ms before taking screenshot
      - taking page screenshot
      - captured a stable screenshot
      - 15 pixels (ratio 0.01 of all image pixels) are different.

Fifteen pixels. The diff image Playwright saved in test-results/ shows them in red, on one character: the seconds in "Ordered at 21:37:40". The page prints the time of the order, and the second ticked over between the two runs.

This is the number one cause of flaky visual tests: content that changes on every run. Times, dates, random ids, ads, live prices, a "3 people are looking at this" counter. The call log also shows Playwright handling the other usual suspects for you: it disables CSS animations, waits for web fonts, and takes screenshots until two in a row match before comparing.

The fix is to mask the changing part. Playwright paints a solid box over it in both screenshots, so it's never compared:

test("checkout looks right", async ({ page }) => {
  await page.goto("/checkout.html");
  await expect(page).toHaveScreenshot("checkout.png", {
    mask: [page.getByTestId("eta")], // the order time changes every run
  });
});

The old baseline has the unmasked time in it, so replace it once, deliberately:

npx playwright test --update-snapshots

tests/checkout.spec.ts-snapshots/checkout-chromium-linux.png is re-generated, writing actual.
  2 passed (1.4s)

Three more runs, a few seconds apart: 2 passed (947ms), 2 passed (966ms), 2 passed (949ms). Stable.

Masking isn't the only option. You can also freeze the input, with a fixed clock (page.clock) or fixed test data, so the page always shows the same thing. That's better when the changing part matters, for example a price you want to see. Mask what you don't care about. Freeze what you do.

Step 5: The regression

Now the realistic part. A developer working on a new menu page wants bigger buttons there, so they're easy to tap. They make a one-line change to the shared stylesheet:

-button { font: inherit; cursor: pointer; }
+button { font: inherit; cursor: pointer; min-width: 120px; } /* big enough to tap on the new menu page */

Run the tests:

  ✓  1 [chromium] › tests/checkout.spec.ts:3:5 › removing the tip updates the total (147ms)
  ✘  2 [chromium] › tests/checkout.spec.ts:10:5 › checkout looks right (563ms)

    Error: expect(page).toHaveScreenshot(expected) failed

      3439 pixels (ratio 0.01 of all image pixels) are different.

  1 failed
  1 passed (1.3s)

The functional test passed: the × button still exists, still has its name, and still removes the tip. The visual test failed, and here are the three images it saved, cropped to the card:

Left: the baseline. Middle: after the change. Right: the diff, with changed pixels in red. The pink bar is the masked order time.

The × button grew to 120 pixels, the row ran out of room, and the tip label wrapped onto two lines, pushing everything under it down. Nothing is "broken" in the functional sense. It just looks broken. This is exactly the bug visual tests exist for.

The fix is to keep the rule where it belongs:

/* Only the menu page wants big buttons. */
.menu button { min-width: 120px; }

2 passed. The baseline didn't change, because the page is back to how it was approved.

Step 6: How different is "different"? We tested it

Two settings decide what counts as a change:

• threshold: how different a single pixel's colour must be to count as changed, from 0 (any difference) to 1 (anything goes). The default is 0.2.
• maxDiffPixels or maxDiffPixelRatio: how many changed pixels you'll tolerate before failing. The default is zero: one changed pixel fails the test.

We changed the Pay button from #15803d to #166534, a darker green. Anyone looking at the two side by side would see it. Here's what each threshold made of it:

So the default lets a real colour change through. That's on purpose: a slightly loose threshold stops tests failing on tiny anti-aliasing differences between machines. But if colour matters to you (brand colours, or contrast you fixed for accessibility), lower it for that test, or test colour directly with toHaveCSS("background-color", ...).

Step 7: When the change is intended

Designs change on purpose too. The designer asks for a pill-shaped Pay button:

-.pay { ... border-radius: 10px; }
+.pay { ... border-radius: 999px; }

      212 pixels (ratio 0.01 of all image pixels) are different.
  1 failed
  1 passed (1.3s)

The test fails, correctly: something changed. Look at the diff, confirm it's the change you wanted and nothing else, then accept it:

npx playwright test --update-snapshots

tests/checkout.spec.ts-snapshots/checkout-chromium-linux.png is re-generated, writing actual.
  2 passed (1.3s)

Commit the new PNG with the CSS change. The reviewer then sees the before and after images in the pull request, next to the code that caused them. GitHub and GitLab both show image diffs. A visual change that nobody looked at is the one thing this whole process exists to prevent.

Common mistakes

• Updating baselines without looking. --update-snapshots accepts every difference, including the bug you were meant to catch. Run it only after checking each diff.
• Making baselines on a laptop and comparing in CI. Fonts and rendering differ between operating systems, so a Mac baseline fails on a Linux CI runner. Generate and compare baselines in the same environment, for example Playwright's official Docker image for your version, both locally and in CI.
• Screenshotting whole pages full of moving parts. Mask what changes, freeze the clock, use fixed test data, or screenshot one element: await expect(page.locator(".card")).toHaveScreenshot(). Smaller screenshots fail for clearer reasons.
• Raising the threshold until it passes. If a test is flaky, find the moving part. A threshold loose enough to hide flakiness also hides real changes, as Step 6 shows.
• Hundreds of screenshots on day one. Start with your most important pages and states: checkout, sign-up, the error state of the main form. Every baseline is something a person has to review when the design changes.

Built-in, or a service?

Playwright's built-in screenshots, as here, are free and live in your repo. Hosted services such as Percy, Chromatic and Applitools add a web page for reviewing and approving diffs, history, many browsers at once, and smarter comparisons that ignore some rendering noise. Teams often start built-in and move to a service when reviewing PNGs in pull requests stops scaling.

What's next

In Part 4 of Testing in Practice we look at testing database changes: what to check when the code writes to a database, and a hands-on run with SQL and pytest.

ProvaLab's QA Infrastructure runs Playwright tests on real browsers in the cloud, the same environment on every run, which is what visual baselines need, and records every session so a failed screenshot comes with a video of what happened. See it in the two-minute product tour.

1. A load test at the expected peak: does it hold up?
2. A stress test past the peak: where does it slow down, and does it break?

Every command and every line of output below was run while writing this post.

Before you start: a load test is, by definition, a flood of traffic. Only ever point it at something you own, and never at production without a plan. Our target is the API from Part 4, running on our own machine.

Step 1: Get k6 (1 minute)

k6 is a single program. Install it from k6.io for your operating system, or run it from its container image, which is what we do here:

docker pull grafana/k6:latest
docker run --rm grafana/k6:latest version

k6 v2.3.0 (commit/e088784614, go1.27.1, linux/amd64)

Step 2: Start the target (1 minute)

In the Part 4 folder, with the fixed version of the API from Part 6:

uvicorn app:app

One request by hand, to know what "normal" looks like before we add load:

curl -s -o /dev/null -w "%{http_code} in %{time_total}s\n" -X POST http://127.0.0.1:8000/discounts/apply \
  -H "Authorization: Bearer token-alice" -H "Content-Type: application/json" \
  -d '{"order_total": 500, "code": "SAVE10"}'

200 in 0.039147s

Forty milliseconds for one request, on an idle machine. The question load testing answers is: what happens to that number when it's fifty requests at once?

Step 3: Write the load test (4 minutes)

Create a folder called k6 and in it load.js:

import http from "k6/http";
import { check, sleep } from "k6";

const BASE = __ENV.API_URL || "http://127.0.0.1:8000";
const HEADERS = { Authorization: "Bearer token-alice", "Content-Type": "application/json" };

export const options = {
  stages: [
    { duration: "30s", target: 50 }, // ramp up to 50 users
    { duration: "60s", target: 50 }, // hold the expected peak
    { duration: "15s", target: 0 },  // ramp down
  ],
  thresholds: {
    http_req_duration: ["p(95)<200"], // 95% of requests under 200 ms
    http_req_failed: ["rate<0.01"],   // less than 1% errors
  },
};

export default function () {
  // 7 in 10 users apply a discount, 3 in 10 look at an order
  if (Math.random() < 0.7) {
    const res = http.post(`${BASE}/discounts/apply`, JSON.stringify({ order_total: 500, code: "SAVE10" }), { headers: HEADERS });
    check(res, { "discount applied": (r) => r.status === 200 && r.json("total") === 450 });
  } else {
    const res = http.get(`${BASE}/orders/42`, { headers: HEADERS });
    check(res, { "order returned": (r) => r.status === 200 });
  }
  sleep(1); // think time: a real user pauses between actions
}

The four ideas in this file are the four ideas in any load test:

• Stages describe the shape of the load over time: ramp up, hold, ramp down. The ramp matters; starting 50 users in the same millisecond tests something different (that's a spike test).
• Thresholds are the pass/fail rules, decided before running: p95 under 200 ms, under 1% errors. Without them you get numbers but no verdict.
• The default function is what one virtual user (VU) does, over and over: here, a realistic mix of two actions.
• Think time (sleep(1)) is the pause a real person takes between actions. Without it, 50 virtual users behave like thousands of real ones.

The check lines are the load test's assertions: a 200 with the right total. A load test that only counts responses would happily report success while every response was an error page.

Step 4: Run it (2 minutes)

docker run --rm --network host -e API_URL=http://127.0.0.1:8000 -v "$PWD/k6:/scripts" \
  grafana/k6:latest run /scripts/load.js

(As in Part 6, on macOS and Windows use http://host.docker.internal:8000 and drop --network host.)

k6 prints a progress line every second, then the summary:

  █ TOTAL RESULTS

    checks_total.......: 4145    39.338342/s
    checks_succeeded...: 100.00% 4145 out of 4145
    checks_failed......: 0.00%   0 out of 4145

    ✓ discount applied
    ✓ order returned

    HTTP
    http_req_duration..............: avg=1.91ms min=975.39µs med=1.68ms max=16.09ms p(90)=2.7ms p(95)=3.15ms
    http_req_failed................: 0.00%  0 out of 4145
    http_reqs......................: 4145   39.338342/s

    EXECUTION
    iteration_duration.............: avg=1s     min=1s       med=1s     max=1.01s   p(90)=1s    p(95)=1s
    iterations.....................: 4145   39.338342/s
    vus............................: 1      min=1         max=50
    vus_max........................: 50     min=50        max=50

default ✓ [ 100% ] 00/50 VUs  1m45s

Passed. 4,145 requests over 1 minute 45, every check green, no failures, and a p95 of 3.15 ms against a target of 200 ms. At its expected peak this API isn't remotely stressed. (The "iteration_duration" of 1 s is just the think time.)

Note the tick next to default and the absence of any threshold error: that's k6's verdict. In CI, a crossed threshold makes k6 exit with a non-zero code, which fails the job.

Step 5: Stress it (5 minutes)

Passing at 50 users tells you it's fine today. It doesn't tell you how much headroom you have, or what failure looks like. For that, keep increasing the load past the expected peak until something gives. Create stress.js:

import http from "k6/http";
import { check } from "k6";

const BASE = __ENV.API_URL || "http://127.0.0.1:8000";
const HEADERS = { Authorization: "Bearer token-alice", "Content-Type": "application/json" };

export const options = {
  stages: [
    { duration: "20s", target: 100 },
    { duration: "20s", target: 300 },
    { duration: "20s", target: 600 },
    { duration: "20s", target: 1000 }, // keep going past the expected peak
    { duration: "10s", target: 0 },
  ],
  thresholds: {
    // we expect these to fail: the point is to see where
    http_req_duration: ["p(95)<500"],
    http_req_failed: ["rate<0.05"],
  },
};

export default function () {
  const res = http.post(`${BASE}/discounts/apply`, JSON.stringify({ order_total: 500, code: "SAVE10" }), { headers: HEADERS });
  check(res, { "status 200": (r) => r.status === 200 });
  // no think time: every virtual user hammers the API as fast as it can
}

Two deliberate changes: the load climbs in steps to 1,000 users, and there's no think time, so each virtual user sends requests as fast as the API answers. Run it the same way, with stress.js:

  █ TOTAL RESULTS

    checks_total.......: 70982   788.671898/s
    checks_succeeded...: 100.00% 70982 out of 70982
    checks_failed......: 0.00%   0 out of 70982

    ✓ status 200

    HTTP
    http_req_duration..............: avg=498.54ms min=731.6µs med=430.55ms max=1.79s p(90)=1.1s p(95)=1.51s
    http_req_failed................: 0.00% 0 out of 70982
    http_reqs......................: 70982 788.671898/s

    EXECUTION
    vus............................: 13    min=5          max=997
    vus_max........................: 1000  min=1000       max=1000

default ✓ [ 100% ] 0000/1000 VUs  1m30s
time="2026-10-29T18:31:28Z" level=error msg="thresholds on metrics 'http_req_duration' have been crossed"

Read it line by line:

• 70,982 requests in 90 seconds, 789 per second. Twenty times the load test's rate.
• Still 0% failed, every check green. The API never fell over. Every one of 70,982 requests got a correct 200.
• But p95 is 1.51 s, and the maximum 1.79 s. One in twenty customers would wait a second and a half for a discount to apply. That's the threshold k6 crossed, and the reason the run is marked failed.

So the honest answer from this stress test is: this API doesn't break under 1,000 hammering users; it slows down, from 3 ms to 1.5 s at the 95th percentile. That's the "breaking point" for a single-process server: the point where the queue of waiting requests grows faster than it drains. Whether that's acceptable depends on your product, which is exactly why thresholds are written down before the run.

What we'd do next

• Find the knee. Re-run with finer steps (200, 250, 300…) to find the user count where p95 first crosses 500 ms. That number is your capacity, and it belongs in the test plan.
• Watch the server, not just k6. CPU, memory and, for a real app, the database. Here the limit is one Python process on one core; uvicorn --workers 4 would roughly quadruple it. Load testing tells you that it slows; server metrics tell you why.
• Run it from a separate machine. We ran k6 on the same laptop as the API, so they competed for CPU. For real numbers the load generator must not share resources with the target, and for big loads it must be several machines.
• Add a soak run: 50 users for two hours, to catch memory leaks a 90-second test never will.

Putting it in the pipeline

Nightly, not on every commit (a load test takes minutes and needs a quiet environment):

      - run: uvicorn app:app &
      - run: >
          docker run --rm --network host -e API_URL=http://127.0.0.1:8000
          -v "$PWD/k6:/scripts" grafana/k6:latest run /scripts/load.js

If the thresholds are crossed, k6 exits non-zero and the job goes red. Over months, that catches the commit that made things slower.

Habits worth keeping

1. Decide the thresholds before you run. Numbers without a target are trivia.
2. Read p95 and p99, never the average. The average in the stress run (498 ms) hid a p95 three times worse.
3. Assert on the response, not just the status. A fast wrong answer is still wrong.
4. Model real behaviour: a mix of actions, with think time. Drop the think time only when you mean to stress.
5. Never load-test what you don't own. And never production without warning the people who run it.

Hands-On Testing: the complete series

That closes the series. Over seven parts you've built and run:

1. UI testing explained · 2. Your first Playwright test
2. API testing explained · 4. Your first API tests
3. Security testing basics: the OWASP Top 10 · 6. Your first security scan with OWASP ZAP
4. Your first load test with k6 (this post)

Together with the 23-part Testing Basics series, that's a complete foundation: the ideas, and the tools in your hands.

ProvaLab's load testing runs k6 tests like these from the cloud, on machines that don't compete with your app, shows p95 and throughput live, and can find the user count where your API first slows down, step by step. Your coding agent can write the script from your code and explain the slow spots afterwards. See it in the two-minute product tour.

This guide takes the three tests from Part 2 and grows them into a suite with ten tests that is cheap to change. Every command below was run as written, against provalab.io, with these versions:

What we're building

playwright-in-practice/
├── pytest.ini
├── conftest.py          # fixtures: saved state, viewport, mocked API
├── pages/
│   ├── __init__.py
│   ├── blog.py          # BlogPage, PostPage
│   └── guide.py         # GuidePage
├── test_blog.py
├── test_consent.py
└── test_guides.py

Four ideas, one each for the four problems above:

1. Page objects so a locator lives in one place.
2. Fixtures so a test asks for what it needs instead of building it.
3. Saved browser state so the cookie banner is dismissed once, not ten times.
4. A mocked API so the form test never creates a real lead.

Then we run the whole thing on four workers.

Setup (2 minutes)

Same folder and virtual environment routine as Part 2, plus the parallel runner:

mkdir playwright-in-practice && cd playwright-in-practice
python3 -m venv .venv && source .venv/bin/activate
pip install pytest-playwright pytest-xdist
playwright install chromium

pytest.ini points at the site and silences one warning:

[pytest]
addopts = --base-url https://provalab.io
filterwarnings = ignore::pytest.PytestRemovedIn10Warning

The second line is worth a sentence. pytest 9 deprecated a pattern that pytest-playwright 0.10 still uses for its base_url fixture, and prints a warning on every run. It is harmless, and the plugin will catch up; until then this keeps the output clean so that a warning you should read doesn't hide among ones you shouldn't.

Step 1: Page objects (10 minutes)

A page object is a small class that knows where things are on one page and what a user can do there. Tests use it instead of locators. Here is the blog index and a post:

# pages/blog.py
import re

from playwright.sync_api import Page


class PostPage:
    def __init__(self, page: Page):
        self.page = page
        self.title = page.get_by_role("heading", level=1)
        self.all_posts = page.get_by_role("link", name="All posts")

    URL = re.compile(r"/blog/[a-z0-9-]+/$")


class BlogPage:
    def __init__(self, page: Page):
        self.page = page
        self.heading = page.get_by_role("heading", level=1)
        self.cards = page.get_by_test_id("blog-card")
        self.pdf_banner = page.get_by_test_id("blog-pdf-banner")

    def open(self):
        self.page.goto("/blog/")
        return self

    def open_first_post(self) -> PostPage:
        self.cards.first.get_by_role("link").first.click()
        return PostPage(self.page)

    def pdf_link(self, series: str):
        return self.pdf_banner.get_by_role("link", name=f"{series} PDF")

Two things to notice. The attributes are locators, not elements: self.cards is a description of how to find the cards, evaluated fresh every time a test uses it, so it never goes stale after navigation. And an action that leads somewhere returns the page object for where you end up: open_first_post() hands back a PostPage.

The second page object covers the free-PDF pages, which have a form:

# pages/guide.py
from playwright.sync_api import Page


class GuidePage:
    """The free-PDF page for one series, e.g. /testing-basics/."""

    def __init__(self, page: Page, slug: str):
        self.page = page
        self.slug = slug
        self.parts = page.get_by_test_id("pdf-part")
        self.download_link = page.get_by_test_id("download-link")
        self.error = page.get_by_test_id("lead-error")

    def open(self):
        self.page.goto(f"/{self.slug}/")
        return self

    def request_pdf(self, name: str, email: str):
        self.page.get_by_test_id("lead-name").fill(name)
        self.page.get_by_test_id("lead-email").fill(email)
        self.page.get_by_test_id("lead-submit").click()

request_pdf is the kind of method page objects exist for: three low-level steps that every test would otherwise repeat, named after what the user is doing.

Now the three tests from Part 2, rewritten. They read like sentences, and there is not a single selector in them:

# test_blog.py
from playwright.sync_api import expect

from pages.blog import PostPage


def test_blog_lists_posts(blog):
    expect(blog.heading).to_have_text("ProvaLab Blog")
    expect(blog.cards.first).to_be_visible()


def test_a_post_opens_from_the_list(blog):
    post = blog.open_first_post()
    expect(post.page).to_have_url(PostPage.URL)
    expect(post.title).to_be_visible()
    expect(post.all_posts).to_be_visible()


def test_pdf_banner_links_to_both_guides(blog):
    expect(blog.pdf_link("Testing Basics")).to_have_attribute("href", "/testing-basics/")
    expect(blog.pdf_link("Hands-On Testing")).to_have_attribute("href", "/hands-on-testing/")

Where does blog come from? That is the next step.

Step 2: Fixtures (5 minutes)

A pytest fixture is a function that builds something a test needs; a test asks for it by naming it as a parameter. You have been using them since Part 2: page is a fixture that pytest-playwright provides. Yours go in conftest.py, which pytest loads automatically:

# conftest.py (first part)
import pytest

from pages.blog import BlogPage


@pytest.fixture
def blog(page) -> BlogPage:
    return BlogPage(page).open()

A fixture can ask for other fixtures. blog asks for page, wraps it and opens the blog. Every test that names blog gets a fresh page, already on /blog/, with no setup lines of its own.

pytest-playwright also lets you override its own fixtures. The most useful one is browser_context_args, the options every new browser context is created with. A fixed viewport and locale make screenshots and date formats the same on every machine:

@pytest.fixture(scope="session")
def browser_context_args(browser_context_args):
    return {
        **browser_context_args,
        "viewport": {"width": 1280, "height": 800},
        "locale": "en-GB",
    }

scope="session" means it runs once per test session, not once per test. Keep that word in mind for the next step.

Step 3: Dismiss the cookie banner once (10 minutes)

Every page on this site shows a small cookie notice until you close it. In Part 2 we ignored it, because none of the three tests clicked anything under it. In a bigger suite it will eventually sit on top of something you need, and the lazy fix, a click at the top of every test, is slow and gets copied everywhere.

Playwright has a better answer. A browser context can save its state (cookies and local storage) to a file, and a new context can start from that file. So: dismiss the banner once, save, and have every test's context begin with it already gone.

# conftest.py (second part)
import os
from pathlib import Path

# One state file per pytest-xdist worker, so parallel runs never write the same file.
STATE = Path(__file__).parent / ".state" / f"consent-{os.environ.get('PYTEST_XDIST_WORKER', 'main')}.json"


@pytest.fixture(scope="session")
def consent_state(browser, base_url) -> Path:
    """Dismiss the cookie banner once per session and save the browser state."""
    if not STATE.exists():
        context = browser.new_context(base_url=base_url)
        page = context.new_page()
        page.goto("/")
        page.get_by_role("button", name="Dismiss").click()
        STATE.parent.mkdir(exist_ok=True)
        context.storage_state(path=STATE)
        context.close()
    return STATE


@pytest.fixture(scope="session")
def browser_context_args(browser_context_args, consent_state):
    """Every test's browser context starts with the banner already dismissed."""
    return {
        **browser_context_args,
        "storage_state": str(consent_state),
        "viewport": {"width": 1280, "height": 800},
        "locale": "en-GB",
    }

consent_state is session-scoped, so the banner is dismissed once per run. browser_context_args now asks for it and passes the file as storage_state. Nothing else in the suite mentions the banner at all. The saved file is small and readable:

{"cookies": [], "origins": [{"origin": "https://provalab.io", "localStorage": [{"name": "cookie-consent", "value": "dismissed"}]}]}

What went wrong the first time

Our first version clicked the banner's OK button instead of Dismiss. The run was green, the state file was {"cookies": [], "origins": []}, and the banner was still there in every test. The home page has two buttons called "OK", and the click had landed on the other one. The lesson is the one from Part 1 again: a locator should describe one thing. "Dismiss" is the banner's only button with that accessible name, so that is the one the fixture clicks.

Two tests prove the state works, and show the difference between a context built from the fixture and a bare one:

# test_consent.py
from playwright.sync_api import expect

BANNER = "We use cookies for analytics"


def test_banner_is_gone_with_saved_state(page):
    page.goto("/")
    expect(page.get_by_text(BANNER)).to_have_count(0)


def test_banner_shows_for_a_fresh_visitor(browser, base_url):
    context = browser.new_context(base_url=base_url)   # no storage_state
    page = context.new_page()
    page.goto("/")
    expect(page.get_by_text(BANNER)).to_be_visible()
    context.close()

The same trick is how you handle login. Sign in once in a session fixture, save the state, and every test starts signed in. Put the state file in .gitignore: it holds whatever cookies the site set.

Step 4: Mock the API so no real lead is created (10 minutes)

The free-PDF pages have a form: name and email, then a download link appears. Behind it the page POSTs to a lead API. A test that fills the form for real creates a real lead every run, and someone has to delete "Test Reader" from the inbox each morning.

page.route intercepts requests inside the browser, before they leave it. The test decides what the server says.

As a fixture, so any test can ask for it:

# conftest.py (third part)
@pytest.fixture
def mock_leads(page):
    """Answer the lead API inside the browser, so tests never create a real lead.
    Returns the list of request bodies the page sent."""
    sent = []

    def handle(route):
        sent.append(route.request.post_data_json)
        route.fulfill(status=201, json={"id": 1})

    page.route("**/api/v1/public/leads", handle)
    return sent

The fixture returns the list it appends to, so a test can check what the page sent, not just what it showed afterwards. Here are the guide tests, parametrised over both PDF pages:

# test_guides.py
import pytest
from playwright.sync_api import expect

from pages.guide import GuidePage

GUIDES = [("testing-basics", 23), ("hands-on-testing", 7)]


@pytest.mark.parametrize("slug,parts", GUIDES)
def test_guide_lists_every_part(page, slug, parts):
    guide = GuidePage(page, slug).open()
    expect(guide.parts).to_have_count(parts)


@pytest.mark.parametrize("slug,parts", GUIDES)
def test_pdf_appears_after_the_form(page, mock_leads, slug, parts):
    guide = GuidePage(page, slug).open()
    guide.request_pdf("Test Reader", "reader@example.com")
    expect(guide.download_link).to_have_attribute("href", f"/downloads/{slug}.pdf")
    assert len(mock_leads) == 1
    assert mock_leads[0]["email"] == "reader@example.com"
    assert mock_leads[0]["source_path"] == f"/{slug}/"


def test_rejected_submission_shows_an_error(page):
    page.route("**/api/v1/public/leads", lambda route: route.fulfill(status=422, json={"detail": "invalid"}))
    guide = GuidePage(page, "testing-basics").open()
    guide.request_pdf("Test Reader", "reader@example.com")
    expect(guide.error).to_be_visible()
    expect(guide.download_link).to_have_count(0)

@pytest.mark.parametrize runs a test once per row of GUIDES, so two tests become four, each with its own name in the output. The last test skips the fixture and routes inline, because it wants a different answer: a 422. That is the real value of mocking. A real server gives you the happy path all day; the sad paths, a rejection, a 500, a timeout, you get on demand only when you are the server.

One rule: mock the things around what you are testing, never the thing itself. Here the subject is the page's behaviour after an answer, so faking the answer is fair. A test of the lead API would have to call the lead API.

Step 5: Run it, then run it in parallel (3 minutes)

The whole suite, ten tests, one browser:

$ pytest -v
test_blog.py::test_blog_lists_posts[chromium] PASSED                     [ 10%]
test_blog.py::test_a_post_opens_from_the_list[chromium] PASSED           [ 20%]
test_blog.py::test_pdf_banner_links_to_both_guides[chromium] PASSED      [ 30%]
test_consent.py::test_banner_is_gone_with_saved_state[chromium] PASSED   [ 40%]
test_consent.py::test_banner_shows_for_a_fresh_visitor[chromium] PASSED  [ 50%]
test_guides.py::test_guide_lists_every_part[chromium-testing-basics-23] PASSED [ 60%]
test_guides.py::test_guide_lists_every_part[chromium-hands-on-testing-7] PASSED [ 70%]
test_guides.py::test_pdf_appears_after_the_form[chromium-testing-basics-23] PASSED [ 80%]
test_guides.py::test_pdf_appears_after_the_form[chromium-hands-on-testing-7] PASSED [ 90%]
test_guides.py::test_rejected_submission_shows_an_error[chromium] PASSED [100%]
============================= 10 passed in 27.14s ==============================

Now the same suite on four workers. pytest-xdist starts four pytest processes and deals the tests out between them; each process gets its own browser and its own session fixtures, which is why the state file is named after the worker:

$ pytest -q -n 4
..........                                                               [100%]
10 passed in 11.56s

Ten tests don't need four workers. A hundred do, and parallel only works if the tests were written as if they would run in any order, each starting from its own goto() and sharing nothing but read-only state. That is why Part 2's habit, "no test depends on another", matters before the suite is big, not after.

Everything from Part 2 still applies on top: --headed to watch, --tracing retain-on-failure to keep a trace of anything red, --browser firefox to run the same ten tests in a second engine.

Habits worth keeping

1. Page objects hold locators and actions, never assertions. The test decides what should be true; the page object only knows where to look.
2. A fixture per recurring need. If two tests share three setup lines, that is a fixture.
3. Session scope for anything expensive and read-only: saved state, a signed-in session, test data that doesn't change.
4. Mock what creates real records, and the sad paths. Everything else, hit for real.
5. Write every test to run alone and in any order. Then -n 4 is free when you need it.

What's next

This suite runs on one laptop and one browser. The next step is to run it on the browsers and devices your users actually have, keep every trace somewhere the whole team can see it, and have failures land with the agent that wrote the code. That is what ProvaLab's QA Infrastructure does: the same Playwright tests connect to real browsers and devices in the cloud, every session is recorded, and a failing test's trace goes straight back to your coding agent through the ProvaLab plugin or MCP. See it in the two-minute product tour.

If you came here without doing Part 2 first, start at Your first Playwright test. The whole Hands-On series, seven parts, is also one free PDF.

Fundamentals

1. What is software testing? Checking that software does what it should, and doesn't do what it shouldn't. A test has a starting point, an action and an expected result; a mismatch is a bug. (Part 1)

2. Can testing prove software is bug-free? No. Testing shows the presence of defects, never their absence. Its job is to find the important bugs before users do, and to report the risk that remains. (The 7 principles)

3. Why is exhaustive testing impossible? The number of possible inputs and paths is astronomically large; a ten-character field alone has quadrillions of values. So we choose tests by risk and by technique. (The 7 principles)

4. What are the levels of testing? Unit (one piece of code), integration (pieces together), system (the whole application, end to end) and acceptance (does it meet the real need). Each catches what the others miss. (Part 2)

5. Explain the testing pyramid. Many fast unit tests at the base, fewer integration tests, and a few slow end-to-end tests at the top. The inverted version, mostly end-to-end tests, is slow and flaky. (Part 2)

6. What is the difference between QA, QC and testing? QA improves the process to prevent defects; QC checks the product to find them; testing is the main QC activity. Job titles blur this, but the definitions don't. (Part 20)

7. Verification vs validation? Verification: are we building the product right, against its specification? Validation: are we building the right product, for its users? A product can pass one and fail the other. (Part 20)

Process and planning

8. What is the STLC? The software testing life cycle: requirement analysis, test planning, test case design, environment setup, execution and closure, each with entry and exit criteria. It runs alongside development, not after it. (Part 4)

9. What are entry and exit criteria? Entry: what must be true before a phase starts, such as "build deployed, smoke test passed". Exit: what must be true to finish, such as "all high-priority tests passed, no open critical bugs". They stop testing starting too early or never ending. (Part 4)

10. What goes in a test plan? Objective, scope in and out, approach, environment, entry and exit criteria, people, schedule, deliverables and risks. The most underrated line is "out of scope". (Part 8)

11. Test plan vs test strategy? A strategy is the organisation's long-term approach; a plan applies it to one release or project. (Part 8)

12. What is risk-based testing? Spending the most testing effort where failure is both likely and harmful. Score each area on likelihood and impact, and test the top-right of that grid first and deepest. (Part 8)

13. What does "shift left" mean? Moving testing earlier: reviewing requirements, writing tests before code, running checks on every commit. Bugs found earlier are cheaper to fix. (Part 23)

Test design

14. What is a test case, and what does a good one contain? A set of steps with specific data that checks one thing and states the expected result. It has an ID, a title, preconditions, test data, steps, an expected result and a link to its requirement. (Part 6)

15. Test scenario vs test case? A scenario is what to test, in one line; a test case is exactly how to test one part of it. One scenario becomes several cases. (Part 6)

16. What is boundary value analysis? Testing at and around the edges of a valid range, because bugs like < instead of <= only show at the boundary. For a 0 to 100 field: -1, 0, 1, 99, 100, 101. (Part 6)

17. What is equivalence partitioning? Grouping inputs the software should treat the same, and testing one value from each group instead of all of them. (Part 6)

18. Black-box vs white-box vs grey-box testing? Black-box tests from the outside, against requirements. White-box designs tests from the code's structure. Grey-box tests from outside with some knowledge of the inside, such as the database or the API. (Part 7)

19. Statement coverage vs branch coverage? Statement: did every line run? Branch: did every decision go both true and false? One test can give 100% statement and only 50% branch coverage. (Part 7)

Bugs and reporting

20. What makes a good bug report? A title that says where, what and when; the environment; the fewest steps to reproduce; expected vs actual result; severity and priority; and evidence. Test: could a stranger reproduce it without asking you anything? (Part 9)

21. Severity vs priority? Severity is how badly the bug breaks the software; priority is how soon it must be fixed. A misspelled brand name on the home page is low severity, high priority. (Part 9)

22. Describe the bug life cycle. New, assigned, open, fixed, ready to retest, retest, verified, closed; or reopened if the retest fails. Side exits: rejected, duplicate, deferred, can't reproduce. (Part 10)

23. Who should verify a fix? Ideally the person who found the bug. They know exactly what was wrong, and they retest on every platform the bug affected. (Part 10)

24. What do you do when a developer says "works on my machine"? Add what the report was missing: exact data, environment, a recording. "Can't reproduce" usually means the report lacked something, not that the bug isn't real. (Part 9)

Test types

25. Smoke vs sanity testing? Smoke: a wide, shallow check of every main feature on each new build, to decide whether it's worth testing at all. Sanity: a narrow, deeper check of one area after a change. (Part 14)

26. Regression testing vs re-testing? Re-testing confirms a specific bug is fixed by running the failed test again. Regression testing runs previously passing tests to check the change broke nothing else. Every fix needs both. (Part 15)

27. What is integration testing, and what are stubs and drivers? Testing that separately built parts work together. A stub stands in for a part that is called; a driver stands in for a part that calls. Top-down integration uses stubs, bottom-up uses drivers. (Part 12)

28. System testing vs end-to-end testing? End-to-end follows one complete user journey; system testing covers those journeys plus non-functional checks of the whole system, in a production-like environment. (Part 13)

29. Name the types of performance testing. Load (expected peak), stress (past the peak, to the breaking point), spike (sudden rush), soak (hours, for leaks), volume (large data), scalability (does adding servers add capacity). (Part 16)

30. Why use percentiles instead of the average response time? The average hides the slow tail. A p95 of 1.8 s means one in twenty users waits that long, even if the average looks fine. (Part 16)

31. What is static testing? Finding defects without running the software: reviews, walkthroughs, inspections and static analysis tools. It catches missing requirements and design flaws that no executed test can. (Part 21)

Runs, coverage and traceability

32. Test case vs test run vs test result? A test case is written once; a run executes a set of cases on a specific build and environment; a result is what happened to one case in one run. (Part 17)

33. What do blocked and skipped mean, and why not count them as failed? Blocked: the test couldn't run because something else is broken. Skipped: deliberately not run. Neither tells you anything about the feature, so they're reported separately. (Part 17)

34. How do you calculate pass rate? Of executed tests: passed ÷ (passed + failed). Of planned tests: passed ÷ all planned. Report both; the gap is the blocked and skipped tests. (Part 17)

35. What is a requirements traceability matrix? A table linking each requirement to its test cases, results and bugs. It shows requirements with no tests, failing requirements, and tests that cover nothing. (Part 11)

36. What kinds of test coverage are there? Requirement coverage, execution coverage, code coverage (line, branch, function) and platform coverage. Each has a formula and none of them means bug-free at 100%. (Part 18)

Automation and tools

37. How do you decide what to automate? Three questions: will it run again and again, is the feature stable, is pass/fail clear? Yes to all three, automate; any no, keep it manual for now. (Part 3)

38. What is a flaky test, and why is it dangerous? One that passes and fails on different runs with no code change. Once people re-run red tests until they pass, real failures get re-run away too. (Hands-On 1)

39. What's the best way to locate elements in a UI test? By role and accessible name first, then test IDs, then visible text; CSS tied to styling and positional XPath last. The locator decides whether the test survives a redesign. (Hands-On 1)

40. Selenium vs Playwright vs Cypress? Selenium: the standard, most languages, widest browser and device coverage, manual waiting. Playwright: fast, auto-waiting, all three browser engines, best debugging. Cypress: best developer experience for JavaScript front-end teams. New projects usually pick Playwright. (Hands-On 1)

41. What is TDD? Write a failing test first, write just enough code to pass it, then clean up. The failing test proves the test can catch the bug. (Part 23)

API and security

42. Why test the API before the UI? API tests run in milliseconds, are stable, start before the screens exist, pinpoint the failing endpoint and cover every client at once. Test the business rules there; keep a few UI tests for journeys. (Hands-On 3)

43. 401 vs 403 vs 404? 401: not authenticated, who are you? 403: authenticated but not allowed. 404: not found. Assert the exact one; an API that returns 404 for someone else's record leaks that it exists. (Hands-On 4)

44. What is the most common serious API security bug? Broken access control: a user reading or changing another user's data. The test is four lines: sign in as one user, request another's record, expect 403. (Hands-On 5)

45. What is the OWASP Top 10? The ten most serious web application risks, updated from real-world data. The 2025 edition leads with broken access control and misconfiguration, and adds supply chain failures. (Hands-On 5)

46. SAST vs DAST? SAST reads the source code without running it, on every commit. DAST attacks the running application from outside, nightly or before release. They find different things. (Hands-On 6)

Agile, CI/CD and you

47. How does testing work in an agile team? Quality is the whole team's job, testing happens inside the sprint, acceptance criteria are the test cases, and regression is automated so people can explore. (Part 23)

48. What runs at each stage of a CI/CD pipeline? Every commit: static analysis and unit tests. Every pull request: integration and API tests. Merge: build and smoke test. Nightly: full regression and performance. Release candidate: system and exploratory testing. Production: smoke test and monitoring. (Part 23)

49. Tell me about a bug you found that the tests missed. Pick one that came from exploring beyond the script: an unusual sequence, an edge case, a state nobody considered. Explain how you found it, how you reported it, and what test you added so it stays fixed. (Part 22)

50. How do you decide testing is done? Against the exit criteria agreed in the plan, never against the deadline alone. Report what was tested, what wasn't, what's still open and the risk that remains. (Part 4)

How to use this before an interview

1. Read the short answers once. Mark any you couldn't explain to a colleague.
2. Follow the link for each marked one; every post is a ten-minute read.
3. For questions 7, 21, 25, 26 and 43, the pairs people confuse, practise saying the difference out loud.
4. Prepare one real story for question 49. Interviewers remember stories, not definitions.

Good luck. The whole Testing Basics series and the Hands-On Testing series are free, and every code sample in them actually runs.

What is an API?

An API (application programming interface) is the set of requests a program can make to another program. In a modern app, every screen talks to the server through an API: when you tap "Place order", the mobile app sends a request like POST /orders with the order details, and the server replies with the result.

Most web and mobile APIs are HTTP APIs, usually following the REST style:

• A request names a resource with a URL path, such as /orders/42.
• A method says what to do with it: GET (read), POST (create), PUT or PATCH (update), DELETE.
• Data travels as JSON, a plain-text format of fields and values.
• The reply carries a status code: 200 means OK, 201 created, 400 bad request, 401 not signed in, 403 not allowed, 404 not found, 500 the server broke.

The API is where the real logic lives. Whether a customer orders from the mobile app, the website or the restaurant dashboard, the same POST /discounts/apply decides what the discount is. A bug there shows up once, for everyone.

What is API testing?

API testing sends requests directly to the API and checks the responses, without any user interface.

A test is still locate, act, assert, with "locate" replaced by "build the request":

1. Build a request: method, URL, headers, body.
2. Send it.
3. Assert on the response: status, headers, body and timing.

In Python with the requests library it looks like this (we'll run tests like it for real in Part 4):

response = requests.post(
    "https://api.example.com/discounts/apply",
    json={"order_total": 500, "code": "SAVE10"},
    headers={"Authorization": f"Bearer {token}"},
)
assert response.status_code == 200
assert response.json()["total"] == 450

No browser, no page load, no waiting for elements. That test runs in a few milliseconds.

Why test the API before the UI

The pattern most teams settle on is the testing pyramid applied to the back end: test the business rules thoroughly at the API level, and keep a few UI tests for the journeys users see. The discount logic gets fifty API tests and one UI test, not the other way round.

A second reason that's easy to miss: APIs are products too. If partners or a mobile team build on your API, a change that breaks it breaks them, whether or not your own website still works.

What to check in every response

1. Status code. 200 for success, the right 4xx for bad input or missing permission, and never a 500. A 500 means the server crashed instead of handling the situation.
2. Headers. The content type is what the client expects (application/json); caching is right for the data; and nothing sensitive leaks in a header.
3. Body shape, the contract. The fields the API promises are present, with the right types: total is a number, not the string "450". Clients break on shape changes even when the values are right.
4. Values, the business rule. 500 with SAVE10 must give total: 450. This is the check that finds the bug that charges every customer the wrong amount.
5. Timing. The response arrived within the budget, say under 500 ms. One slow endpoint makes every screen that uses it slow.

Kinds of API tests

Functional tests check that each endpoint does its job: create an order, read it back, apply a discount. The bulk of your API suite.

Negative tests send wrong input on purpose, and check that the API refuses it cleanly: a missing field returns 400 with a helpful message, an unknown order returns 404, a discount of 150% is rejected. Many real bugs hide here, as we saw in test design.

Authentication and authorisation tests check that requests without a valid token get 401, and that a signed-in customer gets 403 when asking for someone else's order. The second kind, called broken object-level authorisation, is the most common serious API security bug, and it's easy to test: sign in as customer A, request customer B's order, expect 403.

Contract tests check the shape of requests and responses against an agreed definition, usually an OpenAPI (formerly Swagger) document. When several teams or partners depend on the API, contract tests catch a renamed field before it breaks anyone. Tools such as Pact go a step further, letting each client declare exactly what it needs.

Integration tests through the API check that a sequence works end to end: create an order, apply a code, pay, then read the order back and confirm the total. This is integration testing with the API as the entry point.

Performance tests send many requests at once to find the breaking point, as described in performance testing. Load tests are nearly always API tests.

Test data and authentication

Two practical problems every API tester meets:

• Tokens. Most APIs need a key or a bearer token. Tests get one in a setup step (sign in with a test account, or read a test key from an environment variable), then reuse it. Never put real credentials in the test code or the repository.
• Test data. POST /orders creates real rows. Tests should create what they need, use it, and clean up, or run against a test environment that's reset regularly. A test that depends on "order 42 already existing" will fail the day someone deletes order 42.

Tools

If you're starting from nothing, pytest with requests needs the least setup and produces tests that live next to your code.

API testing and the UI tests you already have

The two work together. The fastest reliable UI suites use the API to set the stage: create the test customer and the order through API calls in a few milliseconds, then open the browser only for the part that needs a browser. The sign-up form gets one UI test; the thousand ways an order can be wrong get API tests.

What's next

In Part 4 we build a small API of our own, so there's nothing to sign up for and nothing we don't own, and write a pytest suite against it: functional, negative, authorisation and contract checks, with the real output.

ProvaLab's QA AI Agent runs API test cases on its own and records each request and response against its test case, and AppSec fuzzes your API for the authorisation and input-handling bugs described above. See it in the two-minute product tour.

That gives three approaches: black-box, white-box and grey-box testing.

Black-box testing

In black-box testing you test without looking at the code. You know what goes in, what should come out, and what the requirements say. The inside is a closed box.

This is how users experience software, and it is how most QA testing is done. You need no programming knowledge, and the tests stay valid even when developers rewrite the code inside.

The main black-box techniques:

1. Equivalence partitioning

Split the inputs into groups the software should treat the same, and test one value from each group. A discount field that accepts 0–100% has three groups: below 0, 0 to 100, above 100. (More detail, with a diagram, in Part 6.)

2. Boundary value analysis

Test at and around every edge: -1, 0, 1, 99, 100 and 101 for that discount field. Mistakes like < instead of <= only show up at the exact boundary.

3. Decision table testing

When the result depends on combinations of conditions, list every combination in a table, and turn each row into a test. For example: new customer yes/no × order over 300 yes/no → free delivery yes/no.

4. State transition testing

Some features behave differently depending on what state they are in. A food order moves through clear states:

Draw the states and the allowed moves, then test:

• Every allowed move: placed → accepted, preparing → cancelled, and so on.
• The forbidden moves: can a customer cancel an order that is already out for delivery? They mustn't, and the app must say so clearly.

State transition testing is ideal for orders, payments, sign-in (locked after 3 wrong passwords?), subscriptions and any kind of workflow.

5. Use case testing

Test complete journeys the way a real user would: "A customer searches for pizza, adds two, applies a code, pays by card and tracks the delivery." Each use case has a main flow and alternative flows (the card is declined, the restaurant closes mid-order).

6. Pairwise testing

When many settings combine, testing every combination explodes. Three devices × three payment methods × two languages makes 18 combinations. Most bugs are triggered by one setting, or a pair of settings, so pairwise testing picks a smaller set where every pair appears at least once. Here, 9 tests cover every pair instead of all 18. Free tools such as PICT generate these sets for you.

White-box testing

In white-box testing you design tests by looking at the code itself. You read the logic and make sure every part of it is exercised. It is mostly done by developers, usually as unit tests (see Part 5).

The key white-box idea is code coverage: how much of the code your tests actually ran. Here is a small function from our delivery app:

def delivery_fee(order_total, is_member):
    fee = 40
    if order_total >= 300:
        fee = 0
    if is_member:
        fee = fee / 2
    return fee

Statement coverage

Has every line run at least once? One test is enough here:

def test_big_order_member():
    assert delivery_fee(400, is_member=True) == 0

Both if conditions are true, so every line runs. We measured it with pytest-cov:

Name          Stmts   Miss  Cover
delivery.py       7      0   100%

100% statement coverage, from one test. Sounds complete, but it isn't.

Branch coverage

Has every decision gone both ways? Each if can be true or false, and our single test only ever took the "true" side. Turn on branch coverage and the gap appears:

Name          Stmts   Miss Branch BrPart  Cover   Missing
delivery.py       7      0      4      2    82%   3->5, 5->7

3->5 means "the if on line 3 never skipped straight to line 5": we never tested a small order. Add one test where both conditions are false:

def test_small_order_non_member():
    assert delivery_fee(100, is_member=False) == 40

Name          Stmts   Miss Branch BrPart  Cover   Missing
delivery.py       7      0      4      0   100%

Now every decision has gone both ways. This is why branch coverage is a stronger measure than statement coverage.

Condition and path coverage

• Condition coverage checks each part of a combined condition separately, such as if is_member and order_total >= 300: has each half been both true and false?
• Path coverage checks every route through the code. Our function has two ifs, so four paths: big+member, big+non-member, small+member, small+non-member. Our two tests cover only two of them. The small-order member case (fee 20) has never been tested.

Path coverage grows very fast with real code, so teams usually aim for good branch coverage and add path tests only for critical logic.

One warning: coverage tells you what code ran, not whether the tests checked the right things. A test with no assert still counts as coverage. Treat coverage as a map of what you haven't tested, not as proof of quality.

Grey-box testing

Grey-box testing sits in between. You test from the outside, like a user, but you use some knowledge of the inside (the database design, the APIs, the logs or the architecture) to design sharper tests and check results more deeply.

Examples from our app:

• Check behind the screen: place an order in the app, then check the database. Is the order saved with the right total and status, or did the screen just show success?
• Go around the UI: the app only lets customers choose quantities 1–10, but what does the API accept if you send 1,000 or -5 directly? Knowing there is an API makes this test possible.
• Use the logs: trigger a failed payment and check the server logs for a clear error, with no card details written in plain text.
• Integration points: knowing that orders are passed to the restaurant through a message queue, test what happens when that queue is slow or down.

Grey-box testing is common in integration, API and security testing, where knowing a little about the inside finds bugs a pure black-box tester would never reach.

Experience-based techniques

Alongside the three boxes, testers rely on experience:

• Error guessing: "what usually breaks?" Empty fields, very long text, special characters, double clicks, the back button, a lost network.
• Exploratory testing: learning, designing and testing at the same time, guided by curiosity and a time limit.
• Checklists: a team's growing list of things that have gone wrong before.

Comparing the three

The last row matters most: each approach misses what the others catch. White-box testing can't find a feature nobody built, and black-box testing can't see a branch nobody reached. Good teams use all three.

What's next

In Part 8 we put it together in a test plan: what to test, how, by whom and when.

ProvaLab helps on every side of the box: Test Management keeps your black-box test cases, the coding agent writes and runs unit tests from the code, and API and security testing reach behind the UI. See it in the two-minute product tour.

That journey is the bug life cycle, also called the defect life cycle. Knowing it well means fewer bugs get lost, and nobody wonders "wait, who has this one?"

The bug life cycle at a glance

Every status, one by one

New

A tester (or a user, or an automated test) has reported the bug. Nobody has looked at it yet. This is where every bug starts.

Assigned

A test lead, product owner or developer lead has triaged it: confirmed it's a real bug, set its priority, and assigned it to a developer.

Open (or In Progress)

The developer has started working on it: reproducing it, finding the cause, writing the fix.

Fixed

The developer has changed the code and believes the bug is solved. Good practice is to also add an automated test that would have caught it, so it can't quietly come back.

Ready to retest

The fix has been deployed to the test environment, and the bug is waiting for a tester. A fix that only exists on the developer's laptop can't be retested yet. This status makes that clear.

Retest

The tester follows the original steps from the bug report again, on the new build.

Verified

The retest passed: the bug is gone. The tester also runs a few regression tests around it, to make sure the fix didn't break something nearby.

Closed

The bug is done. Usually the tester closes it after verifying, or it closes automatically when the fix is released.

Reopened

The retest failed: the bug is still there, or it only went away in some cases. The tester reopens it with a comment on what still happens, and it goes back to the developer.

The side exits

Not every bug follows the main path. During triage, a bug can also end up as:

"Can't reproduce" is usually a sign the report was missing something, which is why the habits in Part 9 matter so much.

Who moves the bug?

Each status change belongs to someone. If nobody owns a step, bugs get stuck there:

The rule worth remembering: the person who found the bug should be the one who verifies the fix. They know exactly what was wrong.

One bug, start to finish

Let's follow BUG-231 from Part 9, "Expired discount code accepted at checkout":

1. New (Mon 10:05): the tester reports it, with steps, a recording and a link to test case TC-104.
2. Assigned (Mon 11:30): the team lead sets priority High, because it costs money, and assigns it to a developer.
3. Open (Mon 14:00): the developer finds that the code checks the start date of a discount but never the end date.
4. Fixed (Mon 16:20): the developer adds the missing check and an automated test for an expired code.
5. Ready to retest (Tue 09:00): the fix is deployed to staging.
6. Retest → Reopened (Tue 10:15): OLD10 is now refused on the web, but still accepted in the Android app, which has its own copy of the check. The tester reopens the bug with that detail.
7. Fixed again (Tue 15:00): the check moves to the server, so every app uses the same rule.
8. Retest → Verified (Wed 09:30): refused on web and Android; regression tests for valid codes still pass.
9. Closed (Wed 09:35).

The reopen in step 6 is not a failure of the process. It is the process working. Without a proper retest, half the bug would have shipped.

Keeping the bug tracker healthy

• Triage regularly, ideally every day in busy periods, so no bug sits in "New" for a week.
• Watch for stuck bugs. A bug in "Fixed" for days probably never reached the test environment.
• Watch the reopen rate. If many bugs come back, fixes are being rushed or retests skipped.
• Don't let "Deferred" become a graveyard. Review deferred bugs before each release.
• Comment on every status change. "Reopened: still accepted in Android 2.4.0" is far more useful than a silent change.

Your tool's names may differ

Jira, Azure DevOps, GitHub Issues and other trackers each use their own names and allow teams to customise the flow. Some keep it as simple as To Do → In Progress → Done. The ideas stay the same: someone reports it, someone decides, someone fixes, someone checks, and only then is it closed.

What's next

In Part 11 we connect everything with the requirements traceability matrix: the table that proves every requirement has tests, and shows which ones are failing.

In ProvaLab, issues follow this life cycle with their status, priority and assignee, are linked to the test case and test run that found them, and can be raised straight from a failed automated test. See it in the two-minute product tour.

Exploratory testing is how skilled testers find those bugs. It's the most misunderstood kind of testing: often dismissed as "just clicking around", when done well it is one of the most effective techniques there is.

What is exploratory testing?

In exploratory testing, the tester designs, runs and learns from tests at the same time, using what they just discovered to decide what to try next.

In scripted testing, the thinking happens first (writing the test case) and the running happens later, often by someone else. In exploratory testing the two happen together: you try something, see what the software does, form a new idea, and try that.

Three things make it different from "just clicking around":

1. It has a goal. A written charter says what to explore and why.
2. It is time-boxed. A session lasts a fixed time, typically 60 to 120 minutes.
3. It is recorded. The tester keeps notes of what they tried, what they found and what questions came up, and reports them afterwards.

This structured form is called session-based test management, developed by James and Jonathan Bach.

Scripted vs exploratory testing

They are not rivals. Scripted tests, especially automated regression tests, guard against what you already know can go wrong. Exploratory testing finds what you don't know yet. Principle 5, tests wear out, is exactly why you need both.

A session, start to finish

1. Write the charter

A charter is one or two sentences describing the mission. A useful template, from Elisabeth Hendrickson's Explore It!:

Explore (the target) with (resources or approaches) to discover (the information you want).

For our food delivery app:

Explore discount codes at checkout with unusual sequences and inputs to discover ways the discount can be wrong.

A charter is narrow enough to finish in a session and open enough to leave room for discovery. "Test checkout" is too broad; "apply code SAVE10" is a test case, not a charter.

2. Run the session

Set a timer. Start from the charter, and let each result suggest the next idea. Keep notes as you go, in three groups:

• Ideas tried: so you and others know what was covered.
• Bugs found: with enough detail to write a proper bug report afterwards.
• Questions: things the requirements don't answer, and areas you didn't reach.

When something looks wrong, stop and reproduce it before moving on. A bug you can't reproduce is a bug you'll struggle to report.

3. Debrief

At the end, report briefly: what was covered, what was found, what's left. Then decide the next charter. In the example, the discount bugs on the web suggest an obvious follow-up: the same charter on Android.

Heuristics: where ideas come from

Experienced testers don't rely on inspiration. They use heuristics: rules of thumb that generate test ideas on demand. A few of the most useful:

Vary the input

• Boundaries: zero, one, the maximum, one more than the maximum (see Part 6).
• Nothing: empty fields, missing files, no network.
• Too much: very long text, huge files, thousands of items.
• Strange characters: spaces at the ends, emoji, quotes, <script>, text in another alphabet.
• Wrong type: letters in a number field, a date in the past.

Vary the sequence

• Do things in the wrong order. Pay, then add an item.
• Interrupt: go back, refresh, lose the network, close the app mid-way.
• Repeat: click twice, submit twice, apply the same code twice.
• Undo: add then remove, apply then cancel.

Vary the state and environment

• A new user vs a user with 500 orders.
• Two browser tabs doing the same thing at once.
• Switch language, time zone or currency mid-flow.
• Different devices, screen sizes, slow connections.

Follow the data

• Where does this value go after I enter it? Check the order history, the email, the restaurant's screen. Is it the same everywhere?

Think like different people

• A first-time user who misreads the screen. An impatient user. A restaurant owner. Someone trying to get a discount they shouldn't.

Keeping a personal list of heuristics, and adding to it every time a new kind of bug turns up, is one of the fastest ways to become a better tester.

When to use exploratory testing

• New features, before test cases exist or while they're being written.
• Right after a sprint's work lands, to find problems the stories didn't anticipate.
• Around a bug fix, alongside the re-test and regression run.
• Before a release, on the riskiest areas, after the scripted suite has passed.
• When requirements are thin and scripted tests would mostly be guesses.
• For usability: scripts can't tell you a flow is confusing.

Common mistakes

• No charter. Aimless clicking covers the easy paths and misses the rest.
• No notes. If nobody knows what was explored, the session can't be reported, repeated or built on.
• Only happy paths. The value is in the odd sequences and inputs.
• Treating it as a replacement for regression testing. Exploration doesn't guard against old bugs returning; automated scripts do.
• Only junior testers do it. It benefits most from experience; put your most knowledgeable people on the riskiest charters.
• Not reporting bugs properly. A finding in your notes isn't fixed until it's a bug report in the tracker.

What's next

In Part 23, the last part of Testing Basics, we look at testing in agile teams and CI/CD pipelines: how everything in this series fits together when the team ships every week, or every day.

ProvaLab keeps exploratory findings next to scripted results: a bug found in a session is filed with its steps and evidence, linked to the feature, and visible alongside the automated runs for the same release. See it in the two-minute product tour.

Every command and every line of output below was run while writing this post.

What you need: Python 3.10 or newer and a terminal. Nothing to sign up for.

Step 1: Set up (2 minutes)

mkdir first-api-test
cd first-api-test
python3 -m venv .venv
source .venv/bin/activate      # Windows: .venv\Scripts\Activate.ps1
pip install fastapi uvicorn requests pytest

FastAPI builds the API, uvicorn runs it, requests calls it, pytest runs the tests.

Step 2: Build the API (5 minutes)

Create app.py. It's the discount feature from our food delivery app, as an API:

from datetime import date

from fastapi import Depends, FastAPI, Header, HTTPException
from pydantic import BaseModel, Field

app = FastAPI(title="Discount API")

# Who is who: a bearer token maps to a customer (a real API would check a database)
TOKENS = {"token-alice": "alice", "token-bob": "bob"}

# Discount codes: percentage off and the day they expire
CODES = {
    "SAVE10": {"percent": 10, "expires": date(2099, 1, 1)},
    "OLD10": {"percent": 10, "expires": date(2025, 11, 1)},
}

# Orders and who owns them
ORDERS = {42: {"customer": "alice", "total": 500}, 43: {"customer": "bob", "total": 300}}


def current_customer(authorization: str = Header(default="")) -> str:
    token = authorization.removeprefix("Bearer ").strip()
    if token not in TOKENS:
        raise HTTPException(status_code=401, detail="sign in first")
    return TOKENS[token]


class ApplyDiscount(BaseModel):
    order_total: float = Field(gt=0)
    code: str


@app.post("/discounts/apply")
def apply_discount(body: ApplyDiscount, customer: str = Depends(current_customer)):
    code = CODES.get(body.code.upper())
    if code is None:
        raise HTTPException(status_code=404, detail="code not recognised")
    if code["expires"] < date.today():
        raise HTTPException(status_code=400, detail="this code has expired")
    discount = body.order_total * code["percent"] / 100
    return {"total": body.order_total - discount, "discount": discount, "code": body.code.upper()}


@app.get("/orders/{order_id}")
def get_order(order_id: int, customer: str = Depends(current_customer)):
    order = ORDERS.get(order_id)
    if order is None:
        raise HTTPException(status_code=404, detail="order not found")
    if order["customer"] != customer:
        raise HTTPException(status_code=403, detail="not your order")
    return {"id": order_id, **order}

Two endpoints:

• POST /discounts/apply takes an order total and a code, and returns the discounted total. Unknown codes get 404, expired ones 400, and a total of zero or less is rejected with 422 (FastAPI validates the body for us, because of Field(gt=0)).
• GET /orders/{id} returns an order, but only to the customer who owns it: 401 with no valid token, 403 for someone else's order, 404 if it doesn't exist.

Start it in a terminal and leave it running:

uvicorn app:app

Then make one request by hand, to see what a response looks like:

curl -i -X POST http://127.0.0.1:8000/discounts/apply \
  -H "Authorization: Bearer token-alice" \
  -H "Content-Type: application/json" \
  -d '{"order_total": 500, "code": "SAVE10"}'

HTTP/1.1 200 OK
content-length: 47
content-type: application/json

{"total":450.0,"discount":50.0,"code":"SAVE10"}

Status, headers, body: the three things from Part 3, in that order.

Step 3: Fixtures, so every test doesn't repeat itself (2 minutes)

Open a second terminal in the same folder, with the virtual environment activated. Create conftest.py, where pytest looks for shared setup:

import os

import pytest
import requests

BASE_URL = os.environ.get("API_URL", "http://127.0.0.1:8000")


@pytest.fixture
def api():
    """A session signed in as Alice. Every request through it carries her token."""
    session = requests.Session()
    session.headers["Authorization"] = "Bearer token-alice"
    session.base = BASE_URL
    yield session
    session.close()


@pytest.fixture
def anonymous():
    """A session with no token at all."""
    session = requests.Session()
    session.base = BASE_URL
    yield session
    session.close()

A fixture is setup that a test asks for by name. api gives a session already signed in as Alice; anonymous gives one with no token. The address comes from an environment variable, so the same tests can later point at a staging server: API_URL=https://staging.example.com pytest.

Step 4: The discount tests (3 minutes)

Create test_discounts.py:

def test_valid_code_reduces_the_total(api):
    r = api.post(f"{api.base}/discounts/apply", json={"order_total": 500, "code": "SAVE10"})
    assert r.status_code == 200
    assert r.json() == {"total": 450, "discount": 50, "code": "SAVE10"}


def test_response_matches_the_contract(api):
    r = api.post(f"{api.base}/discounts/apply", json={"order_total": 500, "code": "SAVE10"})
    body = r.json()
    assert r.headers["Content-Type"] == "application/json"
    assert set(body) == {"total", "discount", "code"}
    assert isinstance(body["total"], (int, float))
    assert isinstance(body["discount"], (int, float))
    assert isinstance(body["code"], str)
    assert r.elapsed.total_seconds() < 0.5


def test_code_is_not_case_sensitive(api):
    r = api.post(f"{api.base}/discounts/apply", json={"order_total": 500, "code": "save10"})
    assert r.status_code == 200
    assert r.json()["total"] == 450


def test_unknown_code_is_refused(api):
    r = api.post(f"{api.base}/discounts/apply", json={"order_total": 500, "code": "NOPE99"})
    assert r.status_code == 404
    assert r.json()["detail"] == "code not recognised"


def test_expired_code_is_refused(api):
    r = api.post(f"{api.base}/discounts/apply", json={"order_total": 500, "code": "OLD10"})
    assert r.status_code == 400
    assert "expired" in r.json()["detail"]


def test_zero_or_negative_total_is_rejected(api):
    r = api.post(f"{api.base}/discounts/apply", json={"order_total": 0, "code": "SAVE10"})
    assert r.status_code == 422


def test_missing_field_is_rejected(api):
    r = api.post(f"{api.base}/discounts/apply", json={"order_total": 500})
    assert r.status_code == 422

Notice how the five checks from Part 3 show up:

• Status code in every test, and the exact one: 404 for an unknown code, 400 for an expired one, 422 for bad input. Not "some 4xx".
• Headers and body shape in test_response_matches_the_contract: the content type, exactly the three promised fields, the right types. This is the test that catches a renamed field before it breaks the mobile app.
• Values in the first test: 500 with SAVE10 is 450, full stop.
• Timing with r.elapsed: under half a second.
• Negative cases are four of the seven tests. That's roughly the right proportion.

Step 5: The authorisation tests (2 minutes)

Create test_auth.py:

def test_no_token_means_401(anonymous):
    r = anonymous.post(f"{anonymous.base}/discounts/apply", json={"order_total": 500, "code": "SAVE10"})
    assert r.status_code == 401


def test_customer_can_read_their_own_order(api):
    r = api.get(f"{api.base}/orders/42")
    assert r.status_code == 200
    assert r.json()["customer"] == "alice"


def test_customer_cannot_read_someone_elses_order(api):
    r = api.get(f"{api.base}/orders/43")  # Bob's order, Alice's token
    assert r.status_code == 403


def test_unknown_order_is_404(api):
    r = api.get(f"{api.base}/orders/999")
    assert r.status_code == 404

test_customer_cannot_read_someone_elses_order is the most valuable test in this post. Signing in as one customer and asking for another's data is the single most common serious API security bug, and it takes four lines to check. Note the exact code: 403, not "any error". An API that answered 404 here would be quietly telling an attacker that order 43 exists.

Step 6: Run it (1 minute)

pytest -v

collected 11 items

test_auth.py::test_no_token_means_401 PASSED                             [  9%]
test_auth.py::test_customer_can_read_their_own_order PASSED              [ 18%]
test_auth.py::test_customer_cannot_read_someone_elses_order PASSED       [ 27%]
test_auth.py::test_unknown_order_is_404 PASSED                           [ 36%]
test_discounts.py::test_valid_code_reduces_the_total PASSED              [ 45%]
test_discounts.py::test_response_matches_the_contract PASSED             [ 54%]
test_discounts.py::test_code_is_not_case_sensitive PASSED                [ 63%]
test_discounts.py::test_unknown_code_is_refused PASSED                   [ 72%]
test_discounts.py::test_expired_code_is_refused PASSED                   [ 81%]
test_discounts.py::test_zero_or_negative_total_is_rejected PASSED        [ 90%]
test_discounts.py::test_missing_field_is_rejected PASSED                 [100%]

============================== 11 passed in 0.08s ==============================

Eleven tests, 0.08 seconds. The three browser tests in Part 2 took 10.68 seconds. That's the speed difference from Part 3 made concrete: about a hundred times faster per test.

Step 7: Plant a bug (2 minutes)

Change one line in app.py so the discount is subtracted as a flat amount instead of a percentage:

    discount = code["percent"]  # bug: flat amount, not a percentage

Restart uvicorn and run the tests again:

....F.F....                                                              [100%]
=================================== FAILURES ===================================
______________________ test_valid_code_reduces_the_total _______________________

    def test_valid_code_reduces_the_total(api):
        r = api.post(f"{api.base}/discounts/apply", json={"order_total": 500, "code": "SAVE10"})
        assert r.status_code == 200
>       assert r.json() == {"total": 450, "discount": 50, "code": "SAVE10"}
E       AssertionError: assert {'total': 490...de': 'SAVE10'} == {'total': 450...de': 'SAVE10'}
E         Differing items:
E         {'discount': 10} != {'discount': 50}
E         {'total': 490.0} != {'total': 450}

test_discounts.py:4: AssertionError
_______________________ test_code_is_not_case_sensitive ________________________
>       assert r.json()["total"] == 450
E       assert 490.0 == 450

test_discounts.py:21: AssertionError
========================= short test summary info ============================
FAILED test_discounts.py::test_valid_code_reduces_the_total - AssertionError:...
FAILED test_discounts.py::test_code_is_not_case_sensitive - assert 490.0 == 450
2 failed, 9 passed in 0.06s

Exactly the two tests that check values fail; the nine that check status codes, shape and permissions still pass, because the bug didn't touch those. pytest's diff says precisely what's wrong: discount 10 != 50. Put the line back, and you're green again.

What this suite doesn't cover, and what to add next

• A sequence: create an order, apply a code, read it back. That's integration testing through the API.
• Contract from a document: FastAPI generates an OpenAPI document at /openapi.json; tools like Schemathesis can generate hundreds of tests from it.
• Load: the same endpoints under 200 users at once, as described in performance testing types; we build one with k6 later in this series.
• Running in CI: the same pytest step as in Part 2's workflow, with uvicorn app:app & started first, or a test server fixture.

Habits worth keeping

1. Assert the exact status code. The difference between 401, 403 and 404 is the whole point.
2. Test the contract separately from the values. Shape changes break clients even when the maths is right.
3. Make negative tests at least a third of the suite. That's where the bugs live.
4. Always test "someone else's data". One test, four lines, the most common serious API bug.
5. Never put real tokens in the code. Read them from the environment; use test accounts.

What's next

In Part 5 we move to security testing: the OWASP Top 10 in plain language, and which of those risks the tests you just wrote already touch.

ProvaLab's QA AI Agent runs API test cases like these on its own and records every request and response against its test case; AppSec's API fuzzing sends thousands of malformed and unauthorised requests to find the 403-that-should-have-been and the 500-that-should-have-been-a-400. See it in the two-minute product tour.

We will use Python and pytest, the most popular way to write tests in Python. pytest needs almost no setup, and its tests are plain Python functions.

What you need: Python 3.10 or newer, and a terminal. Check your version with python3 --version (on Windows: python --version). No testing experience required.

Step 1: Set up a project (2 minutes)

Create a folder and a virtual environment: a private space for this project's Python packages, so they don't mix with anything else on your computer.

mkdir first-test
cd first-test
python3 -m venv .venv

Turn the environment on:

# macOS and Linux
source .venv/bin/activate

# Windows (PowerShell)
.venv\Scripts\Activate.ps1

Your prompt now starts with (.venv). Install pytest:

pip install pytest

Step 2: Write the code we want to test (2 minutes)

We will keep using the food delivery app from earlier parts. Create a file called shop.py with one function that adds up an order and applies a discount:

def order_total(prices, discount_percent=0):
    """Add up the prices and apply a percentage discount."""
    if not 0 <= discount_percent <= 100:
        raise ValueError("discount must be between 0 and 100")
    subtotal = sum(prices)
    return subtotal - subtotal * discount_percent / 100

This is exactly the kind of small, important piece of code that unit tests (from Part 2) are made for: if the total is wrong, every customer pays the wrong price.

Step 3: Write your first tests (3 minutes)

Create a second file called test_shop.py:

from shop import order_total


def test_total_adds_up_prices():
    assert order_total([200, 200, 50]) == 450


def test_discount_is_applied():
    assert order_total([200, 200, 50], discount_percent=10) == 405


def test_empty_cart_is_zero():
    assert order_total([]) == 0

That's it. Three things to notice:

• The file name starts with test_, and so does each function. That is how pytest finds your tests automatically.
• Each test checks one thing, and its name says what. When it fails, the name alone tells you what broke.
• assert is plain Python: "this must be true". If it isn't, the test fails.

Each test follows the pattern from Part 1: a starting point (some prices), an action (call order_total) and an expected result (the number after ==).

Step 4: Run the tests (1 minute)

pytest -v

You should see:

collected 3 items

test_shop.py::test_total_adds_up_prices PASSED                           [ 33%]
test_shop.py::test_discount_is_applied PASSED                            [ 66%]
test_shop.py::test_empty_cart_is_zero PASSED                             [100%]

============================== 3 passed in 0.01s ===============================

Three tests, a hundredth of a second. The -v (verbose) flag lists each test by name; without it, pytest prints one dot per passing test.

Step 5: Break something on purpose (3 minutes)

A test you have never seen fail proves very little. Let's plant a bug. In shop.py, change the last line so the discount is subtracted as a flat amount instead of a percentage:

    return subtotal - discount_percent   # bug!

Run pytest again:

collected 3 items

test_shop.py .F.                                                         [100%]

=================================== FAILURES ===================================
___________________________ test_discount_is_applied ___________________________

    def test_discount_is_applied():
>       assert order_total([200, 200, 50], discount_percent=10) == 405
E       assert 440 == 405
E        +  where 440 = order_total([200, 200, 50], discount_percent=10)

test_shop.py:9: AssertionError
=========================== short test summary info ============================
FAILED test_shop.py::test_discount_is_applied - assert 440 == 405
========================= 1 failed, 2 passed in 0.02s ==========================

How to read it:

• .F.: the first and third tests passed (.), the second failed (F).
• The > line shows exactly which check failed.
• assert 440 == 405: the code returned 440, but the test expected 405. A 10% discount on 450 should be 45, not 10.
• test_shop.py:9: the file and line number to look at.

The other two tests still pass, because the bug only affects discounts. That's why small, focused tests are so useful: the failing test's name points straight at the problem.

Now put the correct line back, run pytest again, and you're green.

Step 6: Test many cases at once (2 minutes)

Writing a new function for every number gets repetitive. pytest's parametrize runs one test with many inputs. Add this to the end of test_shop.py:

import pytest


@pytest.mark.parametrize(
    "prices, discount, expected",
    [
        ([100], 0, 100),
        ([100], 50, 50),
        ([100], 100, 0),
        ([99, 1], 25, 75),
    ],
)
def test_order_totals(prices, discount, expected):
    assert order_total(prices, discount_percent=discount) == expected

Each row is its own test: if one row fails, you see exactly which inputs broke.

Step 7: Test that bad input is refused (1 minute)

Good tests check what the code shouldn't do, too. Our function must refuse a discount over 100%:

def test_discount_over_100_is_refused():
    with pytest.raises(ValueError):
        order_total([100], discount_percent=150)

This test passes only if the code raises a ValueError. If someone removes that check one day, this test will catch it.

Run everything:

collected 8 items

test_shop.py::test_total_adds_up_prices PASSED                           [ 12%]
test_shop.py::test_discount_is_applied PASSED                            [ 25%]
test_shop.py::test_empty_cart_is_zero PASSED                             [ 37%]
test_shop.py::test_order_totals[prices0-0-100] PASSED                    [ 50%]
test_shop.py::test_order_totals[prices1-50-50] PASSED                    [ 62%]
test_shop.py::test_order_totals[prices2-100-0] PASSED                    [ 75%]
test_shop.py::test_order_totals[prices3-25-75] PASSED                    [ 87%]
test_shop.py::test_discount_over_100_is_refused PASSED                   [100%]

============================== 8 passed in 0.01s ===============================

Eight automated tests, done. Every time you change shop.py, one command tells you in a second whether anything broke.

Handy pytest commands

Five habits of good tests

1. One behaviour per test, with a name that says what it checks.
2. Test the edges: empty lists, zero, the maximum, and invalid input.
3. Watch every new test fail once, so you know it can actually catch a bug.
4. Keep tests fast, so people run them all the time.
5. Run them automatically on every code change, in a CI pipeline.

What's next

You have written unit tests. In Part 6 we step back to the craft behind every test: how to write good test cases, with a template, examples and the techniques testers use to choose them.

When your test suite grows, ProvaLab runs it for you on every change, on real browsers and devices, keeps every result next to its test case, and lets your coding agent read the failures and fix them. See it in the two-minute product tour.

Everything below was run while writing this post, and the output is pasted in unedited.

What you need: Python 3.10 or newer, a terminal, and about 400 MB of disk for a browser. We'll test provalab.io, our own site. Only ever run UI tests against sites you own or have permission to test; point the examples at your own app when you've got the hang of it.

Step 1: Install Playwright (3 minutes)

Create a folder and a virtual environment, as in Testing Basics Part 5:

mkdir first-ui-test
cd first-ui-test
python3 -m venv .venv
source .venv/bin/activate      # Windows: .venv\Scripts\Activate.ps1

Install the pytest plugin for Playwright, then the browser it will drive:

pip install pytest-playwright
playwright install chromium

The second command downloads a copy of Chromium that Playwright controls. It's separate from the Chrome you browse with, so tests never touch your own cookies or tabs.

Step 2: Point the tests at your site (1 minute)

Create pytest.ini so you don't have to repeat the site address in every test:

[pytest]
addopts = --base-url https://provalab.io

Now page.goto("/blog/") means https://provalab.io/blog/. To test a different site, change this one line.

Step 3: Write three tests (5 minutes)

Create test_blog.py:

import re

from playwright.sync_api import Page, expect


def test_blog_lists_posts(page: Page):
    page.goto("/blog/")
    expect(page.get_by_role("heading", level=1)).to_have_text("ProvaLab Blog")
    expect(page.get_by_test_id("blog-card").first).to_be_visible()


def test_a_post_opens_from_the_list(page: Page):
    page.goto("/blog/")
    page.get_by_test_id("blog-card").first.get_by_role("link").first.click()
    expect(page).to_have_url(re.compile(r"/blog/[a-z0-9-]+/$"))
    expect(page.get_by_role("heading", level=1)).to_be_visible()
    expect(page.get_by_role("link", name="All posts")).to_be_visible()


def test_tour_dialog_opens_and_closes(page: Page):
    page.goto("/")
    button = page.locator("#stage-tms").get_by_role("button", name="Watch the full tour")
    button.scroll_into_view_if_needed()
    button.click()
    dialog = page.get_by_role("dialog", name=re.compile("tour", re.I))
    expect(dialog).to_be_visible()
    page.keyboard.press("Escape")
    expect(dialog).to_be_hidden()

What's going on:

• page is a fixture the plugin provides: a fresh browser tab for every test, closed afterwards. Tests can't leak state into each other.
• Locate, act, assert, as in Part 1. get_by_role("button", name="Watch the full tour") finds the button the way a screen reader would; click() acts; expect(...).to_be_visible() asserts.
• expect waits for you. Each assertion retries for up to 5 seconds until the condition is true. That's why there are no sleep() calls, and why these tests won't flake when the network is slow.
• The third test checks a real interaction: a button opens a video dialog, and Escape closes it. Checking that something disappears is just as important as checking that it appears.

Step 4: Run them (1 minute)

pytest -v

collected 3 items

test_blog.py::test_blog_lists_posts[chromium] PASSED                     [ 33%]
test_blog.py::test_a_post_opens_from_the_list[chromium] PASSED           [ 66%]
test_blog.py::test_tour_dialog_opens_and_closes[chromium] PASSED         [100%]

======================== 3 passed, 1 warning in 10.68s =========================

Three real pages loaded in a real browser, a dialog opened and closed, in under 11 seconds. The browser ran headless (no window), which is the default and what CI uses.

(The warning is a deprecation notice inside the plugin itself, not in your code. It's safe to ignore.)

Watch it happen

To see the browser while the test runs, slowed down so you can follow:

pytest --headed --slowmo 500

Do this whenever a test confuses you. Nine times out of ten you'll see the problem within seconds: a cookie banner over the button, a page that hasn't finished loading, a locator pointing at the wrong thing.

Step 5: Break one on purpose (3 minutes)

As in Part 5, a test you have never seen fail proves little. Create test_broken.py with a wrong expectation:

from playwright.sync_api import Page, expect


def test_blog_heading_wrong_on_purpose(page: Page):
    page.goto("/blog/")
    expect(page.get_by_role("heading", level=1)).to_have_text("ProvaLab Articles")

pytest test_broken.py

    def test_blog_heading_wrong_on_purpose(page: Page):
        page.goto("/blog/")
>       expect(page.get_by_role("heading", level=1)).to_have_text("ProvaLab Articles")
E       AssertionError: Locator expected to have text 'ProvaLab Articles'
E       Actual value: ProvaLab Blog
E       Call log:
E         - Expect "to_have_text" get_by_role("heading", level=1) with timeout 5000ms
E         - waiting for get_by_role("heading", level=1)
E           14 × locator resolved to <h1 class="text-3xl sm:text-4xl lg:text-5xl font-bold text-gray-900">ProvaLab Blog</h1>
E              - unexpected value "ProvaLab Blog"
E
E       Aria snapshot:
E       - heading "ProvaLab Blog" [level=1]

test_broken.py:6: AssertionError
========================= 1 failed, 1 warning in 7.98s =========================

This is a much richer failure than a plain assert, and learning to read it is the core skill:

• Expected vs actual are on adjacent lines. Here the test is wrong, not the site.
• "14 × locator resolved to <h1 …>" says Playwright found the element, checked it 14 times over 5 seconds, and it never had the expected text. So this isn't a timing or locator problem; it's a real difference. If the element had not been found, you'd see "waiting for …" with no resolution, and you'd know to fix the locator.
• The aria snapshot shows the page the way a screen reader sees it. When your locator finds nothing, this list tells you what roles and names actually exist, so you can pick the right one.

Step 6: Record a trace (2 minutes)

A trace is a recording of everything the test did: every action, a screenshot before and after each one, the page's HTML, console messages and network calls. Keep one for every failure:

pytest test_broken.py --tracing retain-on-failure

1 failed, 1 warning in 6.59s

The trace lands in test-results/test-broken-py-test-blog-heading-wrong-on-purpose-chromium/trace.zip. Open it:

playwright show-trace test-results/test-broken-py-test-blog-heading-wrong-on-purpose-chromium/trace.zip

The trace viewer opens in a browser and lets you step through the test, see the page at each action, and inspect what was on screen when it failed. For a failure that only happens in CI, a trace is the difference between fixing it in five minutes and not being able to reproduce it at all.

Two related options, both worth adding to pytest.ini:

[pytest]
addopts = --base-url https://provalab.io --tracing retain-on-failure --screenshot only-on-failure

(--video retain-on-failure records a video too, at the cost of slower runs.)

Step 7: Run it in the pipeline (3 minutes)

The same tests, run on every change by GitHub Actions, with traces saved when anything fails:

name: ui-tests
on: [push, pull_request]

jobs:
  ui:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install pytest-playwright
      - run: playwright install --with-deps chromium
      - run: pytest --tracing retain-on-failure
      - uses: actions/upload-artifact@v4
        if: failure()
        with:
          name: playwright-traces
          path: test-results/

--with-deps installs the system libraries the browser needs on a fresh CI machine. The last step uploads the traces as a downloadable artifact whenever the run is red.

When you need more than one machine can give you, many browsers, operating systems and real phones at once, the same tests connect to a browser grid instead of a local Chromium. The test code doesn't change.

Handy options

codegen is a good way to learn the locator syntax: it records your clicks and suggests role-based locators, which you can then tidy up.

Habits worth keeping

1. Prefer get_by_role and get_by_test_id. The #stage-tms CSS selector in the third test is a reasonable exception: it's an ID set on purpose, and it narrows the search to one section.
2. Never time.sleep(). If you feel the need, you're missing an expect.
3. Each test starts from page.goto() and sets up what it needs. No test should depend on another.
4. Keep traces on failure, always. They cost nothing when tests pass.
5. Run headed when confused, headless everywhere else.

What's next

Our three tests all go through the user interface. In Part 3 we go one layer down: API testing, where the same checks run in milliseconds instead of seconds, and why many teams test the API before the UI.

ProvaLab's QA Infrastructure is the "third place" in the diagram above: your Playwright and Selenium tests run on real browsers and devices in the cloud, every session is recorded, and when a test fails the recording and logs go straight back to your coding agent. See it in the two-minute product tour.

We'll use OWASP ZAP, the most widely used free security scanner, against the discount API we built in Part 4. Then we'll plant a real vulnerability and watch the scanner find it.

Every command and every line of output below was run while writing this post.

Before you start: a DAST scan sends thousands of hostile requests. Only ever point it at something you own, and never at a production system: it will create records, trip rate limits and fill logs. Our target runs on our own laptop.

How a DAST scan works

The scanner has to discover the URLs first. For a website it crawls the pages; for an API it reads the OpenAPI document, which lists every endpoint and parameter. FastAPI generates one for us automatically at /openapi.json, which is why the Part 4 API is a convenient target.

Step 1: Get ZAP (2 minutes, plus a download)

ZAP ships as a container image, so there's nothing to install beyond Docker:

docker pull ghcr.io/zaproxy/zaproxy:stable

It's large (about 3.7 GB), because it carries a full Java runtime and a browser.

Step 2: Start the target (1 minute)

In the first-api-test folder from Part 4, start the API and leave it running:

uvicorn app:app

Step 3: Run the scan (5 minutes)

ZAP includes a ready-made API scan script. One command runs discovery, the passive scan and the active scan, and writes a report:

mkdir -p zap && chmod 777 zap
docker run --rm --network host -v "$PWD/zap:/zap/wrk:rw" ghcr.io/zaproxy/zaproxy:stable \
  zap-api-scan.py -t http://127.0.0.1:8000/openapi.json -f openapi \
  -J report.json -w report.md -I \
  -z "-config replacer.full_list(0).description=auth \
      -config replacer.full_list(0).enabled=true \
      -config replacer.full_list(0).matchtype=REQ_HEADER \
      -config replacer.full_list(0).matchstr=Authorization \
      -config replacer.full_list(0).regex=false \
      -config replacer.full_list(0).replacement=Bearer\ token-alice"

What the pieces mean:

• --network host lets the container reach the API on your machine. (On macOS and Windows, use http://host.docker.internal:8000 instead of 127.0.0.1 and drop --network host.)
• -t …/openapi.json -f openapi tells ZAP to import the endpoints from the OpenAPI document.
• -J and -w write JSON and Markdown reports into the zap folder.
• -I stops the script from returning a failure code on warnings, so it finishes cleanly.
• The long -z part configures ZAP's replacer to add Authorization: Bearer token-alice to every request. Without it, every endpoint would answer 401 and the scanner would test nothing but the sign-in wall.
• If the output ends with "Failed to start the main proxy: Address already in use", something on your machine is using port 8080, which ZAP wants for itself. Add -P 8099 (any free port) after zap-api-scan.py. This happened to us on the third run.

Three minutes later, the end of the output:

Number of Imported URLs: 6
Total of 35 URLs
PASS: Directory Browsing [0]
PASS: Vulnerable JS Library (Powered by Retire.js) [10003]
...
PASS: Remote OS Command Injection (Time Based) [90037]
WARN-NEW: X-Content-Type-Options Header Missing [10021] x 1
	http://127.0.0.1:8000/openapi.json (200 OK)
WARN-NEW: Cross-Origin-Resource-Policy Header Missing or Invalid [90004] x 1
	http://127.0.0.1:8000/openapi.json (200 OK)
FAIL-NEW: 0	FAIL-INPROG: 0	WARN-NEW: 2	WARN-INPROG: 0	INFO: 0	IGNORE: 0	PASS: 118

118 checks passed, 2 warnings, no failures. Both warnings are the same kind of thing: a recommended security header the server doesn't send, flagged on the OpenAPI document itself. ZAP rates them Low. For a tiny API with no browser pages they barely matter, but they're real and the fix is one line of middleware, so they go on the backlog.

(A note on the words: the API-scan script reports every finding as WARN unless you give it a rules file saying which ones should FAIL the build. The actual risk level, High to Informational, is in the report.)

What the scanner cannot see

Remember the most valuable test from Part 4: Alice asking for Bob's order and getting 403. ZAP can't find that bug. It has one token, Alice's, and no way of knowing that order 43 isn't hers. If the API wrongly returned Bob's order, the scanner would see a perfectly healthy 200.

This is the single most important thing to understand about security scanners: they find the generic mistakes, not the ones specific to your business rules. The number-one risk in the OWASP Top 10, broken access control, is mostly found by tests you write. The scanner and your test suite are partners, not alternatives.

Step 4: Plant a vulnerability (3 minutes)

To see what a real finding looks like, let's add two things a team might ship by accident. Append this to app.py:

# ---- two things a real team might ship by accident ----
import sqlite3
import traceback

from fastapi import Request
from fastapi.responses import PlainTextResponse

db = sqlite3.connect(":memory:", check_same_thread=False)
db.execute("create table restaurants(name text, cuisine text)")
db.executemany("insert into restaurants values (?, ?)",
               [("Pizza Palace", "pizza"), ("Curry House", "indian"), ("Sushi Go", "japanese")])


@app.get("/restaurants")
def search_restaurants(q: str = "", customer: str = Depends(current_customer)):
    # the query is built by gluing the user's text into the SQL
    rows = db.execute(f"select name, cuisine from restaurants where name like '%{q}%'").fetchall()
    return [{"name": n, "cuisine": c} for n, c in rows]


@app.exception_handler(Exception)
async def show_errors(request: Request, exc: Exception):
    # "debug mode": send the full stack trace back to whoever asked
    return PlainTextResponse("".join(traceback.format_exception(exc)), status_code=500)

The first is A05 Injection: the search text is glued straight into an SQL statement. The second is A10 Mishandling of Exceptional Conditions (and a bit of A02): any crash sends the full stack trace to the caller.

Restart uvicorn, and try the search with a single quote, the classic probe:

curl "http://127.0.0.1:8000/restaurants?q=%27" -H "Authorization: Bearer token-alice"

Traceback (most recent call last):
  ...
sqlite3.OperationalError: unrecognized token: "'"

One character broke the query, and the server told us exactly which database it uses and what went wrong. That's what an attacker's first probe looks like, and what their first clue looks like.

Step 5: Scan again

Run the same docker run command. This time:

WARN-NEW: A Server Error response code was returned by the server [100000] x 1
WARN-NEW: X-Content-Type-Options Header Missing [10021] x 2
WARN-NEW: SQL Injection [40018] x 1
WARN-NEW: Cross-Origin-Resource-Policy Header Missing or Invalid [90004] x 2
FAIL-NEW: 0	FAIL-INPROG: 0	WARN-NEW: 4	WARN-INPROG: 0	INFO: 0	IGNORE: 0	PASS: 117

ZAP found the injection. The detail is in the report:

[High (Low)] SQL Injection (plugin 40018) x1  CWE 89
   uri:      http://127.0.0.1:8000/restaurants?q=%27 | method: GET | param: q
   attack:   '
   evidence: HTTP/1.1 500 Internal Server Error
   solution: Do not trust client side input, even if there is client side validation
             in place. In general, type check all data on the server side. ...
             use PreparedStatement ... with parameters passed by '?'

[Low (High)] A Server Error response code was returned by the server (plugin 100000) x1
   uri:      http://127.0.0.1:8000/restaurants?q=5%3BURL%3D... | method: GET
   evidence: 500

How to read a finding:

• High (Low) means High risk, Low confidence. ZAP saw a 500 when it sent a quote, which is a strong hint but not proof. Confirm every finding by hand before you act on it. We already did: the curl above reproduced it.
• param: q, attack: ' tell you exactly where and how. That's your bug report's steps to reproduce.
• CWE 89 is the standard identifier for SQL injection, useful for looking up fixes and for compliance reports.
• The server-error finding is the symptom of the same bug plus the debug handler: rated Low on its own, but the stack trace it leaks is a gift to an attacker.

Step 6: Triage

A scan report is not a to-do list. Sort it:

• Fix now: the injection and the stack-trace leak. Confirmed, serious, same root cause.
• Backlog: the two headers. Real, minor, one line each.
• Not applicable: the 22 informational notes that the API returned 404 for things that don't exist, and the 4 about responses not being cacheable. An API is supposed to do both. Mark them as not applicable in your tracker so the next scan doesn't repeat them.

Thirty lines of report become two bugs and two backlog items. The bug report for the injection writes itself: the URL, the parameter, the payload ', the expected result (a normal empty list or a 400), the actual result (a 500 with a stack trace), and the CWE.

Step 7: Fix and re-scan

Two changes. The query takes a parameter instead of glued text, and the debug handler goes:

    rows = db.execute("select name, cuisine from restaurants where name like ?", (f"%{q}%",)).fetchall()

Restart, and the single quote now returns an empty list with a 200, as it should:

curl "http://127.0.0.1:8000/restaurants?q=%27" -H "Authorization: Bearer token-alice"

[]

Scan a third time:

Number of Imported URLs: 7
Total of 36 URLs
WARN-NEW: X-Content-Type-Options Header Missing [10021] x 2
WARN-NEW: Cross-Origin-Resource-Policy Header Missing or Invalid [90004] x 2
FAIL-NEW: 0	FAIL-INPROG: 0	WARN-NEW: 2	WARN-INPROG: 0	INFO: 0	IGNORE: 0	PASS: 118

The injection and the server error are gone; 118 checks pass again.

The two header warnings are still there, as expected; they're on the backlog, and a scanner will keep mentioning them until the header is added. That's the right behaviour: a scan is a regression test for security.

Putting it in the pipeline

The same command runs in CI. Start the API, run the scan against it, and keep the report as an artifact:

      - run: pip install -r requirements.txt
      - run: uvicorn app:app &
      - run: mkdir -p zap && chmod 777 zap
      - run: >
          docker run --rm --network host -v "$PWD/zap:/zap/wrk:rw"
          ghcr.io/zaproxy/zaproxy:stable zap-api-scan.py
          -t http://127.0.0.1:8000/openapi.json -f openapi -J report.json -w report.md
          -z "-config replacer.full_list(0).description=auth -config replacer.full_list(0).enabled=true -config replacer.full_list(0).matchtype=REQ_HEADER -config replacer.full_list(0).matchstr=Authorization -config replacer.full_list(0).regex=false -config replacer.full_list(0).replacement=Bearer\ ${{ secrets.TEST_API_TOKEN }}"
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: zap-report
          path: zap/

Without -I, the job fails when ZAP finds anything new, which is what you want once the backlog items are either fixed or listed in a rules file as ignored. Run it nightly rather than on every commit: a full active scan takes minutes, not seconds.

Habits worth keeping

1. Scan with a signed-in session. An unauthenticated scan tests your sign-in page and nothing else.
2. Confirm by hand before raising a bug. "High (Low)" is a lead, not a verdict.
3. Triage into fix-now, backlog, not-applicable, and record the last pile so it stops reappearing.
4. Keep writing access-control tests yourself. The scanner can't.
5. Never scan what you don't own.

What's next

In Part 7, the last of this series, we go hands-on with load testing: a k6 script against the same API, finding where it slows down and where it breaks.

ProvaLab AppSec runs this kind of scan, and API fuzzing, dependency, secrets, container and code scans, against targets you've verified you own, from the cloud, and presents the findings as a plain-language fix list ordered by severity, with your confirmed false positives remembered between runs. See it in the two-minute product tour.

Most developers have already made the switch. In the Stack Overflow Developer Survey 2025, 84% of developers said they use or plan to use AI tools in their work. But trust went the other way: 46% said they actively distrust the accuracy of what those tools produce.

So teams end up in an awkward place. They write code faster than ever, then either slow down to check it by hand, or ship it and hope.

Why AI-written code needs a different kind of checking

AI-written code is not worse than human code by default. It fails in different ways:

• It looks right. The code is tidy and confident, so reviewers skim it.
• It passes its own tests. If the agent wrote the code and the tests together, the tests often check what the code does, not what it should do.
• It handles the happy path. Empty lists, slow networks, expired sessions and two users at once are where it tends to break.
• There is a lot of it. A reviewer who could read 200 lines a day now gets 2,000.

The answer is not to read every line more carefully. It is to make checking as fast as writing.

A simple routine for every AI-written change

1. Write the test cases before the code

Ask the agent for the test cases first, in plain language: what the feature must do, and what must never happen. Read that list yourself. It is ten lines instead of five hundred, and it is where your judgement matters most.

2. Run the tests somewhere real

Tests that only pass on the developer's laptop prove little. Run them on real browsers and devices, against a running build, the way a user would meet the feature.

3. Keep a record of every run

When a test fails next week, you want to know whether it ever passed, on which build, and what changed. A test case and its run history should live in one place, not in a chat log.

4. Check what tests don't cover

Functional tests don't tell you whether an API leaks data or falls over when fifty people use it at once. Add a security scan and a short load test before anything important ships.

5. Send failures back to the agent

This is the step most teams skip. When a test fails, the agent that wrote the code should get the failure, the logs and a recording, and fix it itself. A person approves the result instead of doing the triage.

Where ProvaLab fits

We built ProvaLab so a coding agent can do this whole routine on its own. Through the Claude Code plugin or MCP, the agent runs your tests on cloud browsers and devices, records every result in test management, scans for security issues, runs load tests, and gets the failures back so it can fix them.

You can see it working in the two-minute product tour, or book a 30-minute demo and we will run it against your own app.

AI made writing code fast. Checking it should be just as fast.

A good bug report is one of the most valuable things a tester produces. This part shows you how to write one.

What is a bug report?

A bug report (also called a defect report or an issue) is a written record of a problem: what went wrong, how to make it happen again, and why it matters.

Its job is simple: let someone who has never seen the bug reproduce it, understand it and fix it, without asking you a single question.

Same bug, two reports

The bad report starts a conversation: which code, which app, what happened? The good one ends it.

The fields of a good bug report

Writing a good title

Most people only ever read the title, in a list of fifty bugs. Use this pattern:

[Where] + [what goes wrong] + [under what condition]

• Bad: "Checkout bug"
• Bad: "Discount doesn't work properly"
• Good: "Expired discount code accepted at checkout; total reduced by 10%"
• Good: "Order history page blank on Android 14 after switching language"

Expected vs actual: the heart of the report

Keep these two side by side and specific. "It doesn't work" is not an actual result. "The total becomes 450 and no error appears" is. If the expected behaviour comes from a requirement or test case, link to it: that shows it's a bug and not a matter of opinion.

Severity vs priority

These two get confused constantly, and they are not the same thing:

• Severity is about the software: how badly does the bug break it? Usually set by the tester.
• Priority is about the business: how soon must it be fixed? Usually set by the product owner or team lead.

They often match, but not always:

Common severity levels:

Attachments that save hours

• Screenshot, with the problem area highlighted.
• Screen recording for anything involving several steps or timing. A short video often replaces a page of text.
• Console log from the browser's developer tools (F12 → Console) for web bugs.
• Network log or HAR file when a server request fails. It shows exactly what was sent and what came back.
• Server or app logs, with the time the bug happened.
• Test data: the account, order number or file you used.

Never attach real passwords, card numbers or customers' personal data. Use test accounts, and blur anything sensitive in screenshots.

Ten habits of great bug reporters

1. Reproduce it first. Try at least twice before reporting.
2. Search for duplicates. Someone may already have reported it.
3. One bug per report. Two problems means two reports, because they may be fixed by different people at different times.
4. Use the fewest steps that still trigger the bug.
5. Find the boundaries. Does it happen in other browsers? With other codes? Only for new customers? Each answer narrows the cause.
6. Report facts, not guesses. "The total becomes 450", not "the discount logic is broken".
7. Keep the tone neutral. No blame, no capitals, no "!!!". Developers and testers are on the same side.
8. Write for a stranger. Assume the reader has never seen this part of the app.
9. Say if it's intermittent. "Happens about 2 in 5 tries" is useful; hiding it wastes time.
10. Keep it updated. If you learn more, add it to the report, not to a chat message that will be lost.

A copy-ready template

Title: [Where] [what goes wrong] [under what condition]

Environment: app version, browser/device, OS, server (staging/production)
Preconditions: account, data or settings needed first

Steps to reproduce:
1.
2.
3.

Expected result:
Actual result:

Severity: Critical / High / Medium / Low
Priority: High / Medium / Low
Reproducibility: e.g. 5/5, or 2 in 5 tries

Attachments: screenshot, recording, console log, network log
Links: test case, requirement, related bugs

What's next

Once a bug is reported, its journey has only just begun. In Part 10 we follow it through the bug life cycle: new, assigned, fixed, retested and closed (or reopened).

In ProvaLab, a failed automated test can raise a bug report with the steps, the error, the screen recording and the logs already attached, linked to its test case and test run. Your coding agent can read it and fix the code. See it in the two-minute product tour.

That agreement is the test plan. In the testing life cycle it is Phase 2, and everything after it follows from it.

What is a test plan?

A test plan is a document that describes the scope, approach, resources and schedule of the testing for a release or project.

It answers six questions:

1. Why are we testing? (the objective)
2. What will we test, and what won't we? (scope)
3. How will we test it? (approach)
4. Where will the tests run? (environment)
5. Who does what, and when? (people and schedule)
6. When do we start, and when are we done? (entry and exit criteria)

A good test plan is written for people, not for a filing cabinet. Developers, product managers and testers should all be able to read it in a few minutes and agree.

Test plan vs test strategy

• A test strategy is the long-term, organisation-wide approach: "We automate all regression tests, every release gets a security scan, and we test on the three most-used browsers." It rarely changes.
• A test plan applies that strategy to one release or project: "For release 2.4 we test discount codes on web and Android from 3 to 7 November."

Small teams often have no separate strategy document; its key rules simply live inside each plan.

The sections of a test plan

Formal templates exist. The IEEE 829 standard was the classic one, and it was later replaced by ISO/IEC/IEEE 29119-3. But most teams use a lighter version of the same ideas:

"Out of scope" is the most underrated line in any test plan. Writing down what you won't test avoids the worst surprise of all: a bug in an area everyone thought someone else had checked.

A one-page example

For most releases, the whole plan fits on one page:

If your plan doesn't fit on one or two pages, ask whether every section is really helping someone make a decision.

Risk-based testing: deciding what matters most

There is never time to test everything equally. Risk-based testing spends the most effort where a failure is both likely and harmful:

For each feature, ask two questions:

• How likely is it to break? New code, complex code, code that changed a lot, or code with a history of bugs.
• How bad is it if it breaks? Money, personal data, sign-in and anything that stops customers ordering.

Put the answers in the plan. They decide the order of testing and where the deepest test cases go, and they tell everyone what was tested lightly if time runs out.

How to write one, step by step

1. Read the requirements, and ask questions (STLC Phase 1).
2. Write the objective in one sentence.
3. List what's in and out of scope, with a reason for each "out".
4. Rank the risks, using the matrix above.
5. Choose the approach: which levels and types, and what to automate.
6. Define entry and exit criteria that can be checked, not "when it looks good".
7. Add people, environment and schedule.
8. Review it with developers and the product owner. A plan nobody agreed to isn't a plan.

Test plans in agile teams

Agile teams rarely write long plans for every sprint. They usually keep:

• A light test strategy for the product, updated a few times a year.
• Testing notes in each sprint: acceptance criteria on every user story, and a short checklist of what is risky this sprint.
• A short release plan for bigger releases: scope, risks and exit criteria.

The questions stay the same. Only the paperwork shrinks.

Common mistakes

• Copying last release's plan without changing the scope or risks.
• Vague exit criteria like "testing is complete". Use measurable ones.
• No "out of scope" section, so gaps stay invisible.
• Writing it alone. Developers know where the code is fragile; product owners know what matters most to customers.
• Never updating it. When the scope changes mid-release, the plan should too.

What's next

In Part 9 we look at what happens when a test fails: how to write a good bug report that developers can act on quickly.

In ProvaLab Test Management, a test plan has these fields built in: objective, test lead, dates, entry and exit criteria, and the features in scope. Test cases attach to the plan, and every test run reports back into it. See it in the two-minute product tour.

What is a test case?

A test case is a set of steps, with specific data, that checks one thing about the software and says exactly what result to expect.

Anyone on the team should be able to pick it up, follow it and get the same answer: pass or fail. If two testers can read the same test case and do different things, it isn't finished.

Test scenario vs test case

These two get mixed up a lot:

• A test scenario is what to test, in one line: "Check that discount codes work at checkout."
• A test case is exactly how to test one part of it: which code, which cart, which clicks, which result.

One scenario usually becomes several test cases: a valid code, an expired code, two codes at once, a code on an empty cart, and so on.

The anatomy of a test case

Example: discount codes in the food delivery app

Notice the mix: one positive case (it works when used correctly) and several negative cases (it fails safely when used wrongly). Most real bugs live in the negative cases, so don't skip them.

Techniques for choosing test cases

You can't test every possible input. These techniques help you pick the few cases that catch the most bugs.

Equivalence partitioning

Group inputs that the software should treat the same way, then test one value from each group. If a discount field accepts 0 to 100, there are three groups: below 0, 0 to 100, and above 100. Testing 50 tells you about the whole valid group; you don't need to test 51, 52 and 53.

Boundary value analysis

Bugs love edges. A developer might write < where they meant <=, and only the exact edge value shows it. So test at and around each boundary:

Six well-chosen values catch most of the mistakes a hundred random ones would.

Decision tables

When the result depends on combinations of conditions, list every combination in a table so none is missed. For free delivery:

Each row becomes one test case.

Error guessing

Use experience to guess where things break: empty fields, very long text, special characters like ' or <, double-clicking a button, going back in the browser, or losing the network mid-payment. Experienced testers keep a personal list of these.

Seven habits of good test cases

1. One test, one purpose. If the title needs "and", split it.
2. A clear, specific title. "Apply an expired discount code", not "Discount test 3".
3. Exact test data. "Code OLD10", not "an expired code".
4. Short steps anyone can follow, without knowing the app.
5. A checkable expected result. "Total stays 500", not "it works correctly".
6. Independent. Each test sets up what it needs, so the order doesn't matter.
7. Linked to a requirement, so you know why it exists and what it covers.

Common mistakes

• Vague expected results like "the page should work fine". Nobody can fail that test.
• Only happy paths. Users mistype, double-click and go back. Test that too.
• Giant test cases with 30 steps. When step 17 fails, the rest are useless.
• Copy-paste duplicates that drift apart over time. Reuse and update instead.
• Never reviewing them. A quick review by another tester catches missing cases early.

What's next

In Part 7 we go deeper into test design techniques: black-box, white-box and grey-box testing, and how each one finds different bugs.

In ProvaLab Test Management, every test case has these fields built in, is grouped into suites, links to its requirement, and keeps the history of every run, manual or automated. See it in the two-minute product tour.

Finding those bugs is the job of integration testing.

What is integration testing?

Integration testing checks that separately built parts of the software work correctly together. The parts might be code modules, services, a database, or an outside system like a payment provider.

Each part may pass all its own unit tests and the whole thing can still fail, because the parts disagree about something: a field name, a date format, a unit of money, or what happens when one of them is slow.

Where integration bugs hide

Here is our food delivery app, broken into its main parts:

Unit tests check the boxes. Integration tests check the arrows.

A classic example: the order service sends the payment amount as 450 meaning rupees, but the payment service expects the amount in paise (hundredths of a rupee), so it charges 4.50 instead of 450. Both services pass their unit tests perfectly. Only a test that runs them together catches the bug.

Other typical integration bugs:

• Different field names or formats: one side sends orderId, the other expects order_id; one sends dates as 29/10/2026, the other expects 2026-10-29.
• Missing error handling: the payment service is down, and the order service crashes instead of showing "payment failed, please try again".
• Timeouts: the restaurant notifier takes 30 seconds and the app gives up after 10.
• Wrong assumptions: the order service assumes the database always returns orders newest first, but nobody told the database that.
• Changes on one side: a team renames a field in their API, and the services that depend on it break.

Approaches to integration testing

When there are many parts, in what order do you connect and test them?

Big bang integration

Connect everything at once, then test the whole thing.

• Good: simple; nothing extra to build.
• Bad: when a test fails, the bug could be in any of the connections, so finding it is slow. And you can't start until every part is finished.

It is only reasonable for very small systems.

Incremental integration

Connect and test the parts a few at a time, so when something fails you know it is in the piece you just added. There are three ways to do this.

Top-down

Start with the top-level part (for example, the order service) and add the parts below it one by one. Parts that aren't ready yet are replaced by stubs.

• Good: the main flow of the application is tested early.
• Bad: many stubs to write; the lowest-level parts are tested last.

Bottom-up

Start with the lowest-level parts (for example, the database layer and the payment client) and work upwards. The parts above that don't exist yet are replaced by drivers.

• Good: the foundations are solid before anything is built on them.
• Bad: the full user flow can only be seen at the end.

Sandwich (hybrid)

Do top-down and bottom-up at the same time, meeting in the middle. It is the most common choice for large systems, because different teams can work in parallel.

Stubs and drivers (and other test doubles)

When you test part of a system, the parts it depends on may not be ready, or may be slow, costly or unreliable (no one wants a test that charges a real card). So we replace them with stand-ins, called test doubles:

Test doubles make tests fast and repeatable. But remember: a test against a stub only proves your code works with the stub. At some point you also need tests against the real thing, or against a realistic copy in a test environment.

Two kinds of integration testing

• Component integration testing checks parts inside your system: the order service with your own database and payment service.
• System integration testing (SIT) checks your system with outside systems: the real payment provider's sandbox, an SMS gateway, a maps API.

System integration tests are usually slower and run less often, because outside systems can be slow, rate-limited or briefly unavailable.

What to check at every integration point

For each arrow in your diagram, check:

1. The happy path: a normal request gets the right response, and data arrives correctly on the other side.
2. Data format and units: names, types, dates, currencies, time zones and character encoding all match.
3. Errors: what happens when the other side returns an error, is down, or sends something unexpected.
4. Timeouts and slowness: the caller waits a sensible time and recovers cleanly.
5. Retries and duplicates: if a request is retried, does the customer get charged twice?
6. Security: the call is authenticated, and no secrets or personal data leak in the messages or logs.

Teams with many services also use contract testing (with tools such as Pact): each service writes down what it expects from the others, and both sides test against that agreement, so a change on one side can't silently break the other.

Integration testing in practice

• Integration tests usually run in the CI pipeline, after the unit tests pass.
• Tools like Testcontainers start a real database or message queue in a container just for the tests, then throw it away.
• API tests (with tools like Postman or pytest with requests) are a natural way to test integration points directly.
• Keep them fewer than unit tests and more than end-to-end tests, as the testing pyramid suggests.

What's next

In Part 13 we move up a level to system testing: testing the whole application, end to end, the way a real user would.

ProvaLab runs integration and API tests on every change, records each result next to its test case, and sends failures back to your coding agent with the request, the response and the logs. See it in the two-minute product tour.

What is manual testing?

In manual testing, a person uses the software and checks the result themselves. They click, type, look and decide whether what they see is right.

There are two main styles:

• Scripted manual testing: the tester follows written test cases step by step. "Open the cart. Apply code SAVE10. Check the total drops by 10%."
• Exploratory testing: the tester explores freely, using experience and curiosity. "What happens if I apply two discount codes? Or change the quantity to minus one?"

Manual testing is quick to start. You need no code, just the app and a sharp eye.

What is automated testing?

In automated testing, code performs the checks. A test script opens the app, does the steps, and compares the result with what was expected, without a person watching.

def test_discount_code_reduces_total(cart):
    cart.add("Margherita pizza", price=200)
    cart.apply_code("SAVE10")
    assert cart.total == 180

Once written, that check runs in a second, as often as you like: on every code change, every night, on every browser. That repeatability is the whole point.

Where each one shines

The key idea: automated tests mostly guard what you already know works. They are excellent at catching regressions, things that used to work and broke. Most new kinds of bugs are found by people, exploring in ways nobody wrote a script for.

Side by side

What should you automate?

Ask three questions about any test:

Good candidates for automation:

• Regression tests: everything that must keep working after each change.
• Smoke tests: "does the app start, can people sign in?" before every release.
• Critical journeys: sign-up, sign-in, checkout and payment.
• Cross-browser and device checks: the same flow on Chrome, Safari, Android and iPhone.
• Data-heavy tests: the same form with a hundred different inputs.
• API tests: fast, stable and easy to check.

Better kept manual:

• Brand-new features still changing every week.
• Usability and design: "is this confusing?" needs a person.
• Exploratory testing: looking for the bugs nobody imagined.
• One-off checks you will never repeat.

Automation isn't free

Two hidden costs catch many teams out:

1. Maintenance. When the app changes, tests that depend on it must change too. A button renamed from "Buy" to "Checkout" can break dozens of tests at once.
2. Flaky tests. A flaky test sometimes passes and sometimes fails without any code change, often because of timing or test data. Flaky tests are dangerous: once people stop trusting red results, they start ignoring real failures too.

Good automation is written like good code: readable, small and reliable. A few trusted tests beat hundreds that nobody believes.

Three myths about automation

1. "Automation will replace manual testers." It replaces repetitive checking, not thinking. Exploring, questioning and judging still need people.
2. "We should automate 100% of our tests." Some tests cost more to automate and maintain than they will ever save. Automate where it pays back.
3. "Once written, automated tests run forever." They need the same care as the product code they test.

Common automation tools

Automated tests usually run in a CI pipeline (continuous integration): every time someone changes the code, the tests run automatically and report the result before the change is merged.

Where AI fits in

AI coding agents can now write automated tests from a plain description, run them and read the failures. That makes automation much cheaper to start. It doesn't change the rules above: you still decide what is worth automating, and a person still explores and judges.

What's next

In Part 4 we put it all in order with the Software Testing Life Cycle (STLC): the six phases of testing, from the first requirement to the release.

ProvaLab runs your automated tests on real browsers and devices, keeps every result next to its test case, and sends failures back to your coding agent with a video and logs. See it in the two-minute product tour.

What is MCP?

The Model Context Protocol (MCP) is an open standard, introduced by Anthropic in November 2024, for connecting AI apps to the tools and data they need.

Before MCP, every AI app needed its own custom integration for every tool. MCP replaces that with one standard plug. A tool is wrapped once as an MCP server, and any app that speaks MCP, an MCP client, can use it. People often compare it to USB-C: one shape of connector for many devices.

An MCP server can offer three kinds of things:

• Tools: actions the model can take, such as "run this test suite" or "create an issue".
• Resources: data the model can read, such as a file or a test report.
• Prompts: ready-made instructions a user can pick, such as "summarise last night's run".

The important part: the model decides which tool to call and when, based on what you asked. You don't click buttons in another dashboard; you ask in plain language, and the agent does the work through MCP.

What is a skill?

MCP gives an agent access to a tool. It does not teach the agent how to use it well. That is what a skill is for.

A skill is a folder with a SKILL.md file: a short name, a one-line description of when to use it, and step-by-step instructions. The agent only sees the name and description until your request matches. Then it loads the full instructions. So you can have many skills installed without filling up the agent's memory.

For example, ProvaLab's triage-failures skill tells the agent: read the actual error text before describing a failure, group failures by cause rather than by test, and flag skipped tests. Without it, an agent with the same MCP access might just list twenty red tests that all come from one broken selector.

What is a plugin?

A plugin is the package that ships it all in one install. A Claude Code plugin can bundle:

• Skills: the know-how.
• Slash commands: shortcuts like /provalab-test.
• MCP servers: the connection, configured in a .mcp.json file.
• Optionally, sub-agents and hooks.

In one line: MCP is the access, skills are the know-how, and the plugin is the box they come in.

"MCP 1.0" vs "MCP 2.0": what changed?

Officially, MCP versions are named by date, not by number. The original specification is 2024-11-05, and it was revised in 2025-03-26, 2025-06-18 and 2025-11-25. The 2026-07-28 revision changed the protocol's core design so much that many people call it the next generation of MCP, or "MCP 2.0". Here, "MCP 1.0" means the original design that ran through 2025-11-25.

The headline change: MCP became stateless. In the original design, a client opened a session with an initialize handshake, got a session id, and the server had to remember every connected client. In the 2026-07-28 revision there is no handshake and no session id. Every request carries its own protocol version and capabilities, so any server instance can answer any request. That makes MCP servers much easier to run at scale, behind load balancers and gateways.

Deprecated features keep working for at least twelve months under MCP's new deprecation policy, so existing servers don't break overnight. Teams that relied on session ids will have some migration work. The details are in the official changelog and the release announcement.

What this means for you as a user: very little changes day to day. You still install a server or plugin and ask in plain language. The difference shows up as faster, more reliable connections, because the servers behind them are simpler to run.

How to use ProvaLab through MCP

ProvaLab's features are exposed as tools on one MCP server at https://mcp.provalab.io/mcp. Running tests, reading results, triaging failures, load tests and security scans can all be driven by your coding agent.

Option 1: the Claude Code plugin (recommended)

The plugin gives you the MCP connection plus the skills and slash commands.

/plugin marketplace add provalab-dev/claude-code-plugin
/plugin install provalab@provalab-marketplace

Then sign in once. The sign-in works on any machine, including SSH sessions and CI, because you approve it in a browser on any device:

curl -fsSL https://plugins.provalab.io/install.sh | sh
provalab-auth login

Restart Claude Code, then check the connection with claude mcp list. You should see the ProvaLab server marked as connected.

Option 2: the claude.ai connector

If you work in claude.ai rather than Claude Code, add the ProvaLab connector instead. It points at the same server and gives you the same tools. Use either the connector or the plugin, not both.

What you can ask

Once connected, just ask. The skills pick up your intent:

• "Run the login tests for the checkout project": runs the suite on ProvaLab's cloud browsers and records the run in Test Management.
• "Generate tests for https://staging.example.com/signup and run them": writes the tests from the live page, then runs them.
• "Why did last night's run fail?": reads the failures, groups them by cause and explains them.
• "Load test our API up to 200 users": writes the load test from your repo, shows the cost, and runs it when you say go.

Or use the slash commands: /provalab-test, /provalab-status, /provalab-results and /provalab-load.

A few tips

• Set your project. If your organization has more than one project, set PROVALAB_PROJECT_ID (and PROVALAB_ORG_ID if you belong to several organizations) so results land in the right place.
• Let the agent fix, but you approve. The real gain comes when failures go straight back to the agent that wrote the code. Keep a person on the final approval.
• Allow the tools by name. Plugin tools are named mcp__plugin_provalab_api__<tool>. Use that pattern in your permission rules if you want to skip repeated approval prompts.

The short version

• MCP is the standard plug that lets AI agents use tools.
• Skills teach the agent how to use those tools well.
• Plugins package both into one install.
• The 2026-07-28 revision ("MCP 2.0") made MCP stateless and easier to scale, without changing how you use it.
• With the ProvaLab plugin, your coding agent can run tests, read results, triage failures and run load tests, all from a plain-language request.

Want to see it working? Watch the two-minute product tour or book a 30-minute demo.

The app worked. It just didn't work fast enough, for enough people at once. Finding that out before your customers do is the job of performance testing.

What is performance testing?

Performance testing checks how fast, stable and scalable a system is under different amounts of use. It is a non-functional type of testing: it doesn't ask "does checkout work?" but "does checkout still work, quickly, when 2,000 people use it at the same time?"

It answers questions like:

• How fast does the app respond with a normal number of users?
• How many users can it handle before it slows down or fails?
• What happens during a sudden rush?
• Does it stay healthy after running for hours or days?
• If we add servers, can we serve more users?

The six types of performance testing

Each type answers a different question, and each has its own shape of load over time:

1. Load testing

Can the system handle the load we expect?

Ramp up to the expected number of users, for example the Friday dinner peak, hold it for a while, and check that response times and error rates stay within your targets. This is the most common performance test, and the one to run before every major release.

Example: 2,000 users browsing menus and placing orders for 15 minutes; 95% of pages must load in under 1 second, with less than 1% errors.

2. Stress testing

Where does the system break, and how does it break?

Keep increasing the load beyond the expected peak until something fails. The goal isn't to pass; it's to learn the breaking point and the failure mode. Does the app slow down gracefully and show a friendly "busy, please wait" message, or does it crash and lose orders? Does it recover by itself when the load drops?

3. Spike testing

Can the system survive a sudden rush?

Jump from a small load to a very large one in seconds, then back down. Real spikes happen: a push notification about a 50% discount, a TV advert, a cricket match ending. Auto-scaling systems in particular need this test, because adding servers takes time and the spike arrives first.

4. Soak testing (endurance testing)

Does the system stay healthy over a long time?

Run a moderate, realistic load for many hours, or even days. Some problems only appear slowly: memory that is never freed (a memory leak), database connections that aren't returned, log files that fill the disk, or caches that grow forever. A one-hour test will never find these.

5. Volume testing

Does the system stay fast with a lot of data?

Keep the number of users normal, but fill the system with a large amount of data: millions of past orders, a customer with 5,000 orders in their history, a restaurant with a huge menu. Queries that are instant on a small test database can be very slow on a full-sized one.

6. Scalability testing

Does adding resources let us serve more users?

Increase the load in steps while adding servers or capacity, and check that the system's capacity grows with them. If doubling the servers doesn't come close to doubling the capacity, something, often the database, is a bottleneck that more servers can't fix.

The six types at a glance

You may also hear capacity testing (finding the maximum number of users that still meets your targets), which sits between load and stress testing.

The metrics that matter

Why averages lie: use percentiles

The average response time hides your slowest users:

That's why performance targets use percentiles:

• p50 (median): half of requests are faster than this.
• p95: 95% of requests are faster than this; 1 in 20 users waits longer.
• p99: 99% are faster; 1 in 100 waits longer.

When thousands of people use your app, "1 in 20" is a lot of unhappy customers. A good target sounds like: "p95 under 1 second, error rate under 1%, at 2,000 users."

How to run a performance test, step by step

1. Set clear goals. Which pages and APIs, how many users, and what response time and error rate count as a pass.
2. Model realistic behaviour. Real users don't all hit "Pay" at the same millisecond. Mix the journeys (browse, search, order) in realistic proportions, and add think time: the pause while a user reads a menu.
3. Use a production-like environment. Same server sizes, same configuration and a realistic amount of data. Results from a tiny test server don't predict production.
4. Prepare test data. Enough test accounts, restaurants and codes that thousands of virtual users don't all fight over the same row in the database.
5. Write the test script. Here is a load test written for k6, a popular open-source tool:

import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  stages: [
    { duration: '2m', target: 200 },  // ramp up to 200 users
    { duration: '5m', target: 200 },  // hold
    { duration: '1m', target: 0 },    // ramp down
  ],
  thresholds: {
    http_req_duration: ['p(95)<1000'], // 95% of requests under 1 second
    http_req_failed: ['rate<0.01'],     // less than 1% errors
  },
};

export default function () {
  const res = http.get('https://staging.example.com/api/restaurants?q=pizza');
  check(res, { 'status is 200': (r) => r.status === 200 });
  sleep(1); // think time
}

6. Run it, and watch the servers too. Response times tell you that it's slow; CPU, memory and database graphs tell you why.
7. Find the bottleneck, fix it, and run it again. Performance work is a loop.

Only ever test systems you own or have permission to test. Sending heavy load to someone else's website or API can look like an attack, and can take their service down.

Common bottlenecks

• Slow database queries, often a missing index.
• The "N+1" problem: loading a list of orders, then making a separate database call for each one.
• Too few database connections in the connection pool.
• No caching of data that rarely changes, like menus.
• Slow or rate-limited outside services, such as a payment provider or a maps API.
• Large pages and images that take long to download on mobile networks.

Common mistakes

• Only looking at averages instead of p95 and p99.
• Testing from an overloaded load generator. If the machine sending the traffic is at 100% CPU, you're measuring it, not your app.
• Unrealistic scenarios: no think time, every user doing the same thing, or a test database with ten rows.
• Testing once, just before launch. Run load tests regularly, so you see when a change makes things slower.
• Ignoring errors because response times looked good. Fast errors are still errors.