Testing and Verifying the Python SDK

Overview

Relevant for SDK contributors and anyone deciding how far to trust the SDK

Each of the SDK's claims is enforced by a test or tool in the repository. Offline tests prove every op builds the right request, and a coverage test proves every documented endpoint has an op. The live suite records every call against a real tenant, and a report proves every op returned a 2xx or has a documented reason it can't. An end-to-end scenario then checks that AlertOps behaves as configured.

Key facts:

• Offline unit suite — 1,012 tests, no network, run in minutes.
• Index coverage — every endpoint in spec/index.md and spec/new-ui-index.md maps to an op.
• Live coverage — the latest report (docs/LIVE_COVERAGE.md) shows 784 ops: 656 exercised live with a 2xx, 128 exempt with a documented reason, 0 gaps.
• Behaviour scenario — tools/e2e_scenario.py verify: 13/13 checks passed (run 2026-11-03 23:54).

Before you start

Relevant for Contributors setting up a test environment

Install the development extra into a virtual environment:

python -m venv .venv && . .venv/bin/activate     # Windows: .venv\Scripts\activate
pip install -e ".[dev]"

This adds pytest, respx (HTTP mocking) and pytest-xdist. The live suite, the scenario tool and the sweeper also need a test tenant API key in the repository's .env file:

# .env (never commit this file)
ALERTOPS_API_KEY=your-test-tenant-api-key

Offline unit suite

Relevant for Contributors checking a change before pushing

python -m pytest tests/unit

All HTTP is mocked with respx, so the suite never touches a real tenant. On the run used for this article, 1012 passed.

Index coverage test

Relevant for Contributors adding or renaming ops

tests/unit/test_coverage.py parses spec/index.md (more than 700 endpoint rows) and spec/new-ui-index.md into (method, base, normalised path) keys, with placeholders and numeric segments normalised to {}. Every key must match an op's path or one of its aliases.

python -m pytest tests/unit/test_coverage.py -q

A missing endpoint fails the test with <n> index endpoints have no SDK op: followed by a per-module list of the missing method and path. The only exclusions are rows that aren't AlertOps API endpoints (static assets, third-party telemetry, the WordPress help-center proxy).

Live end-to-end harness

Relevant for Contributors running or writing live tests

The live suite in tests/live/ is skipped unless you opt in with ALERTOPS_LIVE=1. It reads the key from the repository .env, which wins over any exported key.

ALERTOPS_LIVE=1 python -m pytest tests/live
ALERTOPS_LIVE=1 python -m pytest tests/live/test_groups.py -q   # one module

In PowerShell, set $env:ALERTOPS_LIVE = "1" first. The harness supports pytest-xdist (-n 4): names stay unique per worker.

Rules every live test follows

Harness helpers

The make_* helpers use raw ao.request calls so shared setup doesn't depend on the module under test; the tests themselves use SDK resources.

Notification action links need a real token from an alert email, so that test is opt-in:

ALERTOPS_LIVE=1 ALERTOPS_ACTION_LINK='<link from a real AlertOps alert email>' \
  python -m pytest tests/live/test_action_links.py

Live coverage report

Relevant for Contributors proving an op works against the real API

Every live call is recorded as {op, method, url, status} to tests/live/_out/calls-<RUN_ID>-<worker>-<pid>.jsonl (set ALERTOPS_RUN_ID to group sessions). tools/live_coverage.py matches each recorded URL back to an op by path template, so calls made through ao.request count too, and writes the report:

python tools/live_coverage.py                                   # docs/LIVE_COVERAGE.md
python tools/live_coverage.py --module groups --out build/cov_groups.md
python tools/live_coverage.py --run final,acklink,newui        # only these run ids

It prints a one-line summary: ops=… ok=… exempt=… gaps=… unmatched=… -> <out>. Each op lands in one bucket:

Exemptions

An op that can't get a 2xx on the test tenant needs an entry in tools/live_exemptions/<module>.py:

EXEMPT = {
    "groups.access.add_admins": "400 'Nothing to update': every eligible user already has implicit access "
    "(...) and admins_available returns [] on this tenant (verified with App Admin, Group Admin, Basic "
    "and a custom role)",
}

A reason must state the verified status or the safety rule that forbids the call. "Didn't try" is not a reason. Typical reasons in the report:

End-to-end behaviour scenario

Relevant for Anyone who needs proof that SDK-built configuration behaves correctly in AlertOps

tools/e2e_scenario.py proves the result is right: it builds an on-call stack through the SDK (mostly the semantic layer), fires real alerts, and checks timing and side effects.

python tools/e2e_scenario.py build      # create the "SDK-E2E" stack (idempotent)
python tools/e2e_scenario.py verify     # fire alerts, check behaviour, write build/e2e_report.md
python tools/e2e_scenario.py teardown   # delete everything recorded in tools/e2e_state.json

build creates four @example.com users (Alice, Bob and Carol rotate; Mona is the manager), the group SDK-E2E NOC with a 24x7 rotation, the SDK-E2E NOC Escalation policy (Primary at 0, Secondary at +2 min, Manager at +4 min), a workflow, an API integration, a technical service, a maintenance window, a heartbeat, a response play, a topic and an internal status page. Ids go to tools/e2e_state.json, so a re-run reuses them and teardown deletes exactly those. verify waits on real notifications, so it takes more than ten minutes.

The objects can also be inspected in the AlertOps UI. The policy's own timeline preview matches what check 2 read back and what checks 5–7 measured:

Figure 1. The policy built by the scenario, as the UI previews it: Alice (Primary) at 0 m, Bob (Secondary) at 2 m and Mona (Manager) at 4 m, the same ladder the verify step measured in real notifications.

Figure 2. The alert from the escalation timing check (checks 5–8): Closed by the verify step, owned by Alice, routed through SDK-E2E NOC Escalation and the SDK-built technical service from source SDKE2EMonitoring. The timeline lists Alice, Bob and Mona, and the stats show 3 notified by email. Carol, who is off shift, isn't among the recipients.

The sweeper

Relevant for Contributors cleaning up after an interrupted run

If a live run is killed before teardown, tools/sweep.py finds leftovers. It lists every object kind the suite creates (20 kinds, from users to status-page service groups) and matches names that start with the prefix.

python tools/sweep.py                                 # dry run: list leftovers
python tools/sweep.py --delete                        # delete them
python tools/sweep.py --prefix sdktest-11031455       # one run only

It's a dry run unless you pass --delete, and it refuses any prefix that doesn't start with sdktest. Kinds that can't be deleted with an API key (locations) are listed but never deleted.

Add a new endpoint

Relevant for Contributors extending the SDK

1. Declare the Op in src/alertops/resources/<module>.py: method, path, base=, envelope=, paginate=, flags= (DESTRUCTIVE, SIDE_EFFECT, BILLING, NON_JSON, LEGACY) and doc=. Add other documented spellings as aliases=.
2. Run python -m pytest tests/unit -q. The coverage test must pass, and test_all_ops.py picks up the new op automatically.
3. Write a full lifecycle in tests/live/test_<module>.py (create, read back and assert, update, delete, confirm gone) and run it with ALERTOPS_LIVE=1. Fix the op against the real response and record surprises under "Gotchas (verified live)" in the module docstring.
4. Run tools/live_coverage.py --module <root>. If the op can't get a 2xx, add a verified reason to tools/live_exemptions/<module>.py. Gaps must be 0.
5. Regenerate the reference with python tools/gen_reference.py.

# src/alertops/resources/groups.py (illustration of the Op declaration style)
class Groups(Resource):
    governance = Op("GET", "/groups/{group_id}/governance",
                    doc="Where the group is used: escalation_policy[], integrations[], workflows[], ...")

Best Practices

Relevant for Contributors keeping the SDK trustworthy

Live tests

• Register cleanup on the line after the create, before any assertion that could fail.
• Assert the fields you set on read-back, not just a 2xx. The API reference comes from UI code and is sometimes wrong; the live API wins.
• Use the SDK resource under test, not ao.request, except for setup that belongs to another module.
• Run tools/sweep.py (dry run) after any interrupted session.

Exemptions

• Write the observed status and message into the reason, and re-check exemptions when the tenant's plan changes.

Configuration Examples

Relevant for Starting templates for common test runs

Example 1 — Full release verification

ALERTOPS_LIVE=1 ALERTOPS_RUN_ID=release python -m pytest tests/live
python tools/live_coverage.py --run release
python tools/e2e_scenario.py build
python tools/e2e_scenario.py verify
python tools/e2e_scenario.py teardown
python tools/sweep.py

Example 2 — A minimal live test

from tests.live.conftest import uniq

def test_group_round_trip(ao, cleanup):
    g = ao.groups.create(group_name=uniq("example"))
    cleanup.add("delete group", ao.groups.delete_if_exists, g.group_id)   # right after create
    assert ao.groups.find(uniq("example"))["group_id"] == g.group_id
    ao.groups.update_basic(g.group_id, group_name=uniq("example2"), description=["d"])
    assert ao.groups.get(g.group_id).group_name == uniq("example2")
    assert ao.groups.delete_if_exists(g.group_id) is True
    assert ao.groups.delete_if_exists(g.group_id) is False

Troubleshooting

Relevant for Anyone debugging the test tooling

Every live test is skipped

• ALERTOPS_LIVE=1 isn't set in the shell that runs pytest. The skip reason is "live tests need ALERTOPS_LIVE=1".
• The .env file has no ALERTOPS_API_KEY: the ao fixture skips with "no ALERTOPS_API_KEY in .env".

test_coverage fails with "index endpoints have no SDK op"

• A documented endpoint has no op. Add the op, or add the documented spelling to an existing op's aliases=.

The coverage tool reports gaps

• The op has no 2xx in the recorded runs (check --run didn't leave out the run that exercised it). Exercise it in the live test, or add a verified reason to tools/live_exemptions/<module>.py.

The scenario's escalation checks fail

• The checks accept Bob between 1.5 and 4 minutes and Mona between 3.5 and 7 minutes after Alice. If notifications were slow on the tenant, re-run verify.
• verify raises "alert did not open" when no new Open alert from SDK-E2E Monitoring API appears within 90 seconds. Run build first, and check that the integration still exists.

Leftover sdktest-* objects on the tenant

• A run was interrupted before teardown. Run tools/sweep.py to list them, then tools/sweep.py --delete.

Related Articles

• Getting Started with the AlertOps Python SDK
• Calling the API from Python: Parameters, Pagination and Errors
• Async Client and Composable Flows in the Python SDK
• Planning On-Call Setups with the AlertOps Semantic Layer
• Working with Alerts in the AlertOps SDKs

Need help?

Email support@alertops.com or contact your AlertOps representative.