Purpose
The test suite separates deterministic source checks, JavaScript units, browser behaviour, automated accessibility checks, ZIP inspection, and manual review. Passing one layer does not imply the others passed.
Test layers
| Layer | Command | What it checks |
|---|---|---|
| Python source checks | npm test | Claim rules, asset metadata, local-link logic, prompts, workflow, examples, research mapping, and licences through Python unit and release scripts. |
| JavaScript units | npm run test:unit | Escaping, safe links, Markdown rendering, generated document routing, raw-source download boundaries, normalisation, filtering, manifest data, project names, generated files, and approval state. |
| Asset composition | npm run test:compose | All 100 modules resolve the kernel and referenced specialist controls exactly once, without missing or duplicate IDs. |
| Evaluation fixtures | python3 -S scripts/run_evals.py --dry-run --check | All 40 prompts have cases, five conditions compose, frozen v2.1 baselines exist, and prompt-size output matches the checked-in record. No model is executed. |
| Recorded evaluation scoring | python3 -S scripts/score_evals.py --input <recorded-run> | Disclosures, raw outputs, complete criteria, two independent reviews per case-condition, aggregates, and the finite-suite release gate. Requires externally generated outputs. |
| ZIP integrity | npm run test:zip | CRC-readable archives, UTF-8 content, duplicate or unsafe path rejection, and empty-set rejection. |
| Browser behaviour | npm run test:e2e | Homepage, analytics-consent, privacy, static-catalogue, rendered-guide, rendered-prompt, and builder workflows in Chromium, Firefox, and WebKit, including keyboard modal behaviour and extracted ZIP content. |
| Automated accessibility | npm run test:a11y | Axe checks across twelve representative page states in Chromium, Firefox, and WebKit plus DOM assertions. |
| Rendered documents | npm run test:docs | Every maintained document has a current styled HTML route, internal maintained-document links use HTML, and raw Markdown remains an explicit download. |
| SEO artefacts | npm run test:seo | Rendered documents, the static 100-asset index, sitemap, and project crawler guidance reproduce from their sources without drift. |
| Local links | npm run test:links | Repository-relative Markdown and HTML references and local fragments. External URLs are counted but not fetched. |
| Secret detection | Security gates / secret-scan | Gitleaks 8.24.3 scans pushed or proposed commits with redacted output; a scheduled run scans the complete Git history weekly. GitHub secret scanning and push protection remain separate controls. |
| Full local release gate | bash scripts/verify-release.sh | All configured deterministic and browser gates in release order. |
Reproduce CI locally
Use Node.js 22.18.0 and Python 3:
npm ci
npx playwright install chromium firefox webkit
npm audit --audit-level=high
bash scripts/verify-release.shThe browser server is started by Playwright on 127.0.0.1:4178. Tests use deterministic fictional fixtures and must not use personal, production, or secret data. The local release script does not install or execute Gitleaks; the pinned remote security job is a separate required result.
Coverage boundaries
The Chromium, Firefox, and WebKit matrix does not cover every browser version, operating system, screen reader, locale, high-contrast mode, or input device. The local Markdown renderer supports the maintained repository content and is not a general CommonMark implementation. Analytics tests intercept the Google script request and prove the local consent controller's request and state transitions; they do not prove deployed data arrival, property configuration, downstream processing, or legal suitability. Gitleaks and GitHub secret scanning recognize known patterns and heuristics; neither proves that arbitrary content contains no credential or confidential information. The link checker does not make network requests. Sitemap checks prove syntax and source alignment, not search-engine discovery or indexing. The evaluation dry run proves fixture and composition integrity only. Unit tests prove the repository implementations under test, not the quality of model output or generated downstream systems.
Add failure cases whenever a parser, validator, router, renderer, path boundary, or public claim changes. Do not replace behavioural assertions with file-count checks.