Golden-cohort gate (#179)¶

Before-versus-after comparison of genotyping output across the local test cohort, executed to verify that behaviour-changing commits in #179 preserve genotype stability.

Current routing policy (2026-08-11, #233). The mixed-layout refusals recorded in milestone-4 runs are historical. Equal R1/R2 plus singleton/other reads now consume every non-empty FASTQ exactly once under one Kestrel sample; unequal or one-sided mates remain invalid. Run 7's issue #233 comparison.json records 32 base cases plus 10 repeat/derived cases, 42 total, without rewriting earlier run evidence. This page also records the final gated candidate SHA and retained comparison artifact digests.

Every run selected as an attestation of record is registered in the table below. Completed superseded executions are not separate attestations: the preliminary issue #233 run at 49b0cc6 was replaced by run 7 at the final executable candidate. A verdict is intended to attest one candidate commit and nothing after it, so read the run whose candidate matches the tree you are judging.

How candidate commits are recorded. For runs 1 to 4, candidate revisions are operator assertions: harness 1.0.0 did not record git rev-parse, check clean working trees, or accept expected commit SHAs. Each side stored filesystem paths that change across branch checkouts. From harness 1.1.0 forward, starting with run 5, each execution records its HEAD, active branch, and working-tree status across vntyper/, docker/, and scripts/. The comparator rejects recorded revisions that diverge from --expect-before-sha or --expect-after-sha, and --require-clean aborts if uncommitted edits exist. Run 5's candidate commit is a recorded cryptographic fact; runs 1 to 4 rely on operator logs. Run 5's baseline (4fd638a) predates this harness change, so only its candidate side records verified revision metadata.

Runs 1 to 3 evaluate branch #179 against baseline 2fcc6e3. Runs 4 and 5 evaluate branch fix/issue-181-197-followups against baseline 4fd638a (the merge-base with main and release 2.0.6). Run 5 supersedes run 4 as the attestation of record for that branch: candidate ec67fff differs from the tip because Phase-5 fixes and review commits landed subsequently. Runs 1 to 3 remain authoritative records for their respective candidate commits.

No run attests branch tips as permanent properties: tips advance while commit SHAs are immutable. To verify whether a working tree matches a recorded run:

git diff --stat <candidate>..HEAD -- vntyper/ docker/

If the diff is empty, the checkout matches the tested codebase. Against run 5's candidate at release 2.0.7, only vntyper/version.py differs (a version string reported in logs that does not alter branching logic). scripts/ is excluded because it contains test harnesses and validation scripts unimported by the pipeline.

The genotype verdict is PASS in runs 1 to 6. Every genotype field, Confidence label, and Flag matches byte-for-byte between baseline and candidate across all samples and assemblies. Run 6 confirms this quantitatively: exactly two columns were added, the only cells modified were within a new column, and no genotype field was touched anywhere. Run 7 introduces deliberate reachability updates: it demonstrates that 42 previously rejected input combinations execute successfully and losslessly, meaning no successful baseline outputs exist for direct comparison.

Runs 4 and 5 document non-genotype differences explicitly. In runs 1 to 3, all differences were restricted to report presentation. Run 4 includes two delta classes (a version-probe command string and cohort export file structure). Run 5 includes two deltas (the version-probe string and pruned internal working columns). Each difference maps to a specific commit and leaves genotype calls untouched.

Run 5 shares run 4's baseline and case count, but incorporates verified commit metadata and updated export deltas as detailed in the Run 5 section.

Method¶

Prerequisite: install reference assets¶

Every gate execution reads a shared reference/ directory across both worktrees. Install and verify the bundle once from the published release:

vntyper install-references -d reference --references hg19 hg38 GRCh37 GRCh38 hg19_ensembl hg38_ensembl

Symlink the populated reference/ directory into both baseline and candidate trees. The cohort contains all BAM fixtures in tests/data/ across supported assemblies: 7 multi-reference samples across six assemblies (hg19, hg38, GRCh37, GRCh38, hg19_ensembl, hg38_ensembl), their original hg19 subsets, and the hg38 regression case example_40cf. Five cases run without --fast-mode to exercise unmapped read extraction, three run --extra-modules advntr, two run cohort CRAM inputs, and one purpose-built CRAM runs both scan strategies. Run 6 is the first to include CRAM fixtures. Fourteen additional cases alias GRCh37/GRCh38 BAMs under hg19_ncbi/hg38_ncbi identifiers introduced in milestone 5.

The evaluation matrix contains 78 cases (64 for run 7, 60 for run 6, and 58 for runs 1 to 5). Figures across early runs reflect the 58-case matrix.

CRAM fixtures are generated dynamically (scripts/make_cram_fixtures.py). Fresh checkouts derive 72 cases (50 base, 5 non-fast, 3 adVNTR, 14 alias repeats) until CRAM files are synthesized. Two selected cohort CRAMs and the indexed-safe fixture run in both indexed and stream modes, adding six cases to reach 78.

Outputs compared per case: full kestrel_result.tsv rows and headers keyed on Motifs/POS/REF/ALT/Variant, pre-filter kestrel_pre_result.tsv, output_adVNTR_result.tsv when adVNTR executes, coverage_summary.tsv, HTML screening summaries, recorded pipeline steps, exit codes, and executed subprocess command strings.

Run 4 derives the 50 base cases from tests/data dynamically. Five non-fast cases, three adVNTR cases, and three probes follow declared policy (scripts/golden_cohort/matrix.py). Run 4 also introduces four vntyper cohort evaluations (cohort_multi, cohort_multi_pseudonymized, cohort_single, cohort_empty), diffing exports (cohort_kestrel_{csv,tsv,json}, cohort_advntr_{csv,tsv,json}), rendered tables, and pseudonymization outputs.

The CRAM group (#188)¶

make cram-fixtures (scripts/make_cram_fixtures.py) generates lossless CRAM files matching cohort BAMs in tests/data/cram/. Decoded records match source BAMs exactly. Fixtures are built with no_ref=1 because cohort BAM headers lack M5 checksums, exercising container decoding, .crai indexing, and unmapped read extraction without external reference dependencies. Dedicated fixtures in unit and integration suites validate explicit reference resolution.

All CRAM cases execute without --fast-mode to force unmapped read extraction.

Package resolution verification¶

To prevent editable install collisions where sys.meta_path resolves to the developer checkout regardless of process CWD, each test launches through a wrapper that configures PYTHONPATH and verifies that vntyper.__file__ resides inside its own tree. The wrapper asserts marker module presence: vntyper.scripts.pipeline_guards (runs 1 to 3) or vntyper.scripts.cohort_rules (run 4) must be absent on baseline and present on candidate checkouts. All gate runs pass this verification.

Result: run 1, candidate 7344c62¶

kestrel_result.tsv, kestrel_pre_result.tsv, output_adVNTR_result.tsv, and coverage_summary.tsv match identically after removing timestamped ## headers.

Calls match across all assemblies and both checkout sides:

Result: run 2, candidate 1792345¶

Same baseline and 58-case matrix. Re-executed to incorporate production commits d144505 (adVNTR compound repair), c51052c/b4059ce (assembly validation), 7e58eb8 (shell quoting), and 2c92096 (--output-name checking).

adVNTR calls reproduce run 1 exactly across all cases (VID 25561): a5c1 (RU 2, P=6.78e-07), b178 (RU 4, P=3.83e-56), and dfc3 (D17_2&D18_2&D19_2&D20_2&D21_2, RU 2,2,2,2,2, flagged Polymorphic_Call).

Assembly verification: the assembly guard returned agree across all 58 candidate cases (20 hg19, 9 hg38, 8 GRCh38, 7 each GRCh37, hg19_ensembl, hg38_ensembl) with zero conflicts.

Result: run 3, candidate 8537a61¶

Incorporates 2ae28c5 (shlex.quote applied across subprocess boundaries).

Command string comparisons: 480 commands diffed per side. Quoting was byte-identical across all reached commands because filenames contained no shell metacharacters. Differences were restricted to set -o pipefail; prefixes on sort pipes and header inspection order. Cross-match styling on dfc3_hg19_advntr changed from summary-positive to standard summary-box styling to reflect non-matching results (2aa095a).

Result: run 4, candidate ec67fff¶

Branch fix/issue-181-197-followups compared against 2.0.6 release baseline 4fd638a. Covers 58 per-sample cases, 3 probes, and 4 cohort cases (65 runs per side, 130 total).

Attributed deltas: 1. executed_commands differed across 61 cases due to removing a duplicated -h flag in Kestrel help probes (2873ad3). Total commands matched at 1,111 per side. 2. cohort_* exports removed leaked internal tracking columns __row_result and __unified (90f61fa). Category counts and summary tables remained identical.

The evaluation instrument: harness 1.0.0 versus 1.1.0¶

Harness 1.1.0 introduced cryptographic verification of revisions, strict matrix bounds, and md5 checks on unparsed sidecars (pipeline_info.json, output_R1.fastq.gz, cross_match_results.tsv). Re-evaluating run 4 outputs with harness 1.1.0 confirms identical outcomes while preventing silent skips on reduced matrices.

Correction: adVNTR output columns in runs 2 and 3¶

Early drafts stated that Insertion_len was checked in output_adVNTR_result.tsv. Insertion_len is an internal variable used in frameshift calculation that is not written to disk. The 0/3 delta metric applies to public table columns (VID, Variant, NumberOfSupportingReads, MeanCoverage, Pvalue, RU, POS, REF, ALT, Flag).

Adjudication of differences¶

• D1 (Screening emphasis, 5527a49): Negative cases lost erroneous summary-positive CSS styling. Wording remained unchanged.
• D2 (Screening sentence on dfc3_hg19_advntr, 77d590b): Added 15 missing rule mappings in report_config.json, replacing a negative fallback with an accurate description of high-precision Kestrel detection and flagged adVNTR calls.
• D3 (Assembly validation, 078a6c4): Correctly identified assemblies across all 58 cases without raising false rejections.
• D4 (adVNTR Repeat_Unit_7, 52f822e): Inactive in cohort because no sample carries RU 7.
• D5 (Compound adVNTR parsing, a7c3d9e, d144505): Retains greedy token parsing while constraining string splitting, preserving genotype output.
• D6 (Pipefail enforcement, 331ea95): Subprocess pipes safely exit on upstream errors.

Run 5: 4fd638a -> 9816f86¶

Evaluates nine commits resolving adversarial review findings for PR #199 using harness 1.1.0.

• Baseline: 4fd638a (v2.0.6)
• Candidate: 9816f867c28f on fix/issue-181-197-followups, verified clean
• Executions: 65 per side; package resolution verified
• Verdict: DELTAS (all deltas attributed; zero genotype changes)

Deltas: single command replacement on Kestrel version probes (D6) across 61 cases; cohort export columns pruned of internal fields (29 columns vs 31 in Kestrel, 13 vs 15 in adVNTR).

Run 6: cb593b6 -> 48f97fe, milestone 2¶

Re-attested against 60-case matrix (including 50 derived CRAM fixtures).

Quantitative verdict: exactly two columns added across runs (flag_filter_pass and coverage_qc). The only cell values modified resided in flag_filter_pass. No genotype field changed. Corrected mean coverage satisfied the exact mathematical relation mean_new == mean_old * (1 - pct_old / 100) across all 61 cases (#171).

Run 7: 4678851 -> 19c8acd, issue #233¶

Harness 1.4.0. Clean detached worktrees on both sides. Evaluated 64 pipeline cases, three probes, and four cohort sets. The candidate met 67/67 pipeline/probe outcomes and 4/4 cohort outcomes without timeouts, aborted runs, or unverified launches.

The comparison verdict is BLOCKED because baseline 4678851 intentionally refused 42 mixed-layout FASTQ declarations that candidate 19c8acd processes losslessly. Exactly 25 cases ran successfully on both sides, with identical genotype calls and exit codes. Command deltas across 23 cases were restricted to process-substitution FIFO descriptors (/proc/<pid>/fd/5) and temporary path noise.

Candidate execution confirmed exact lossless routing across 40cf's 93 singleton reads, b178's singletons, all three adVNTR successes, CRAM streams, and forced-indexed fail-closed guards. This page also records the final gated candidate SHA and retained comparison artifact digests:

Run 8: c74e9e5 -> 74fcbe0, adVNTR 2.0.x and real --threads (#259)¶

Harness 1.4.0. 78 pipeline cases, 3 probes, 4 cohort sets (85 runs per side). Package resolution verified. Distinguishing marker: vntyper.modules.advntr.advntr_genotyping:resolve_advntr_threads.

Both sides executed installed adVNTR 2.0.2 with identical references. The gate isolated thread inheritance: baseline ran adVNTR at -t 1 while candidate ran at -t 8.

Genotype outputs: 0 deltas across advntr_result, kestrel_result, coverage_summary, screening_summary, and all cohort artifacts. Executed command deltas in 76 cases were limited to /proc/<pid>/fd/N process noise; the 3 adVNTR cases demonstrated -t 8 parameter delivery without genotype deviation.

Artifact digests: - result.json: 3979e774ccaed31e8877f7c2441980849a59a1b44b5fb95bcf6a4082cb992751 - result.md: a62a0fa5d6064ba6fc4c64c1719c3d3f0b55a0ed30540a7f50f7f769eff943a3 - Baseline side.json: 5321f9206405b77ccf4a198603b1ad1ca1b2f628105aba7a38f8cadaeaddb075 - Candidate side.json: c8f583664a48615116573697ce1d3114ec45a13482fdf183d7c5d2e757f79335 - Matrix snapshots: 241aeaf5aa64b8f684848f271e9d1a45e422ba41825adda9ee67b6a32cd7ac68

Run 9: 80ac6be -> edaf44a, atomic BAM and BAI installation (#314)¶

Harness 1.5.0. 85 runs per side over 78 pipeline cases. Distinguishing marker: vntyper.scripts.artifact_publish.

Baseline wrote output BAM and BAI files directly to public target paths. Candidate writes to deterministic sibling .partial paths (<path>.partial), validating zero exit status and non-empty file creation before atomic os.replace publication. On error, .partial files are unlinked, preventing corrupt intermediates from populating results.

Verdict: IDENTICAL under --expect-command-delta. All genotype and QC artifacts match. Command string deltas in 79 of 81 cases reflect .partial target names and process descriptor noise.

Artifact digests: - result.json: 3c84f5970a0a145e13116bc44ec46951e225cdbee283b4ae6dc76f0878f54b7b - result.md: 13344703ac581b5ca75aca6eaa8d8ef1d98786f5b6f8e5b43e31e2d3b2305ef6 - Baseline side.json: 424e33c894b032987cd1971a1b79dfb44c5916c1cf4d4603857cd0b0346d1148 - Candidate side.json: 00bedbf8d8fb1f24b78d91d44b29915dd018c91a7897ca7d9a7627127eb676f9 - Matrix snapshots: e3a0509d30d646f836eb129b79edded4960b5890673b2e31cc0f89835933e940

Run 10: a632aa1 -> 936f11e, reporting floor split and profile revision 2 (#311)¶

Harness 1.5.0. 85 runs per side over 78 pipeline cases. Distinguishing marker: vntyper.scripts.confidence_assignment:HAS_REPORTING_FLOOR_SPLIT.

Candidate splits the numerical reporting floor from the Low_Precision boundary (0.00469), mandates gg_depth_score_threshold, and updates the packaged decision profile to revision 2 (0b13d07370491b3ea773e65144891cb30caebcae70b0ef98feb0f2c5ccd2f4a1).

Because the numerical floor remains 0.00469, all genotype calls, scores, confidence tiers, and summary tables match baseline outputs. Observed deltas are limited to metadata fields in cohort exports (Decision_Profile_Revision: '1' -> '2') and command string process noise.

Artifact digests: - result.json: 51fe52a3f6b49e7b45e82ad8e09952571b9ef763c445c0113e2a177308f17ed1 - result.md: 235122e7c5c8defcac1e74496a06efc75f71b12873f1449b86e74ea71217f0b1 - Baseline side.json: 5ae4bdc4c3607ff7f08815c16db8beb78f877ca528fa8124b624ea58f4dcaf5e - Candidate side.json: 8596731beb38755b33470bf3def84e6e899e26833444b9699b9769f7ca3fca8f - Matrix snapshots: d218de77d9db2a7802015a1c76aedd51fd4c61c6dddaef87dc489cde3b738339

Run 11: a0d27b5 -> 2cf4946, derived confidence grade and report masthead chip (#173)¶

Harness 1.5.0. 85 runs per side over 78 pipeline cases. Distinguishing marker: vntyper.scripts.screening_summary:supports_confidence_grade.

Candidate calculates derived sample-level confidence grades and displays a corresponding chip in the HTML report masthead (summary_report.html) under state_chips.

All variant calls, depth scores, confidence tiers, flags, TSVs, BED files, and summary sentences match baseline outputs. Command string differences reflect ephemeral process IDs and temporary path noise.

Verdict: IDENTICAL under --expect-command-delta.

Artifact digests: - result.json: fb9565a222b7ebdd6175ef58216537572c0032ab46afb534a4a60c4a67044100 - result.md: da72d53de325f4516e63f7b85b92aaa3813bbd3c5b1d398b3d292539afd871c6 - Baseline side.json: 5852493a1ddb374e5dce2f059eec4aa3555c80595a71f916c8e717d30798a83b - Candidate side.json: 4b897bd131fc82197d45110dd198fcb33f2e6568e5a2f014e61616fbdb8bca95 - Matrix snapshots: bf8555cd7e734cf8e723e80423ecdbd30e9f83459d8d5e4a42313b63ec82881f