Conformance & Testing

Every PurRDF engine is gated by its official test suite. The suites are vendored and byte-frozen in-repo — never hand-edited, SHA-256-verified on every make check so a silent content edit fails the build. The full, live scoreboard is docs/CONFORMANCE.md; this chapter explains how the machine works.

The single conformance matrix

The native Rust W3C harnesses and the Python rdflib drop-in gate are reported together as one scoreboard:

make conformance    # aggregates every suite into one table

The aggregator runs each suite in a fixed order and prints per-suite pass / xfail-or-skip / fail counts with an overall RED/GREEN verdict. It exits non-zero on any unexpected failure, and the rendered matrix in docs/CONFORMANCE.md is itself drift-guarded in CI (the gate fails if the committed block is stale).

What is gated

At the time of writing every suite is green — for example 1,105/1,105 attempted shexTest validation cases, 129/129 W3C SHACL, 264/264 codec round-trips, 70/70 W3C SPARQL entailment-regime cases, and 258/262 agreeing verdicts on the vendored W3C OWL 2 DL-consistency corpus — with the remaining non-passes strictly ledgered (five SPARQL fixtures with upstream-errata non-canonical XSD lexicals; 4 typed OWL 2 divergences). Two of those numbers need their scope stated: the OWL 2 DL corpus is a subset, 262 of the 482 consistency-shaped cases W3C published, and rule-table coverage is not entailment conformance — on this vendored W3C corpus of OWL 2 RL entailment tests the chase scores 50/50, being 27 of 27 positive and 23 of 23 negative, the latter meaning no unsoundness was found. Always read the current numbers from docs/CONFORMANCE.md rather than this snapshot.

Ledger discipline

A harness never skips silently. Four mechanisms keep the scoreboard honest:

1. Exact totals — each harness asserts the number of discovered tests, so corpus drift fails loudly.
2. XFAIL ledgers — every known gap is listed with a reason string, and the harness asserts it still fails. Fixing a gap without removing its ledger entry breaks the build (XPASS discipline), so the ledgers double as roadmaps.
3. Trait skips (ShEx only) — whole spec features can be skipped by manifest trait tags, counted exactly. The list is currently empty.
4. A monotone budget (ratchet) — a committed baseline records the exact allowed count of ledgered gaps per suite. A larger live count fails RED (regression), and a smaller one also fails RED until the budget is lowered to lock the gain in. The budget may only ever be edited downward, which makes “the skip list only shrinks” a mechanical guarantee rather than a convention.

Running the suites locally

make conformance                                     # the single matrix
cargo test -p purrdf-shex                            # all four ShEx suites
cargo test -p purrdf-shapes --test w3c_conformance   # W3C SHACL scoreboard
cargo test -p purrdf-sparql-conformance              # W3C SPARQL
cargo test -p purrdf-rdf                             # RDFC-1.0 + codec goldens
make projection-oracles                             # W3C CSVW + independent CSVW/OBO checks
cargo test -p purrdf-gts                             # GTS vectors

make check — the full local gate (fmt, clippy, build, tests, hygiene) — runs the Rust suites as part of the workspace gate.

Frozen means frozen

The GTS vectors in vectors/ are shared byte-exact with the sibling GTS engines in other languages and are never regenerated in this repository; the wire format is governed in gmeow-gts. The same never-hand-edit rule applies to every vendored corpus and everything under generated/.