Changelog¶

All notable changes to VNtyper 2 are documented on this page.

Unreleased¶

No unreleased changes.

2.0.47 (2026-10-09)¶

Concordance follows the reconciled name¶

• Resolved disagreements count as concordant: since 2.0.45 a disagreement of description is reconciled to the allele both callers saw, but the report still printed Concordance: No match beside the agreed name. The chip came from the cross-match alone, which compares each caller's raw inserted or deleted bases in its own frame. It now also reads Match when reconciliation named adVNTR's allele on a Kestrel row at Tier A or B without caller-disagreement, and the grade is Finding, corroborated accordingly. The sentence then reads "Kestrel and adVNTR name one allele after reconciliation; their raw records are written differently."
• Unchanged: cross_match_results.tsv, result rows, tiers and thresholds. A sample whose callers still disagree keeps No match.

2.0.46 (2026-10-08)¶

CRAM runs record the header step¶

• Header parsing for CRAM input (#342): --cram runs now record the BAM Header Parsing step from the header the assembly guard already read. The step was recorded for BAM input only, so every CRAM report printed Assembly detected from BAM header: not recorded by this run, the aligner warning was never evaluated, and vntyper cohort showed N/A for assembly and alignment pipeline. A CRAM summary now starts with BAM Header Parsing, and fastq_bam_processing/pipeline_info.json is written for CRAM as for BAM.
• Integration compatibility: observation sets accept summary_steps_overrides, which replace a case's recorded step list and nothing else. The three CRAM contracts frozen since 2.0.11 are unchanged; the 2.0.46 observation states their new step list.
• Research pipeline decisions: No caller thresholds, result rows, tiers or screening messages changed.

2.0.45 (2026-10-08)¶

Allele names decided by the resolved haplotype¶

A Kestrel result row is one edit of a haplotype written on one of many motif pairs, so it does not state which sequence was observed. The deepest haplotype record in output.bam does, and reconciliation now reads it. See The resolved haplotype decides the sequence.

• Tier A is reachable again: a name both callers agree on reaches Tier A when Kestrel's deepest haplotype spells the canonical unit carrying it. motif-context-diverges and the gates unioned over equivalent representations then no longer apply, because they describe other pair references. On the development simulation 90 names reach Tier A, all 90 exact; through 2.0.44 none did.
• Disagreements of description are named: when the callers disagree and that haplotype spells the canonical unit carrying the adVNTR name and not the Kestrel name, the reconciled name is adVNTR's at Tier B, without caller-disagreement, with a note stating the reason. Each caller's own name stays beside it. On the development simulation 26 disagreements resolve this way, all to the simulated allele.
• Measured projection: 154 displayed, 136 exact, 18 wrong becomes 162, 151 and 11 on the 200 simulated carriers; no control gains a finding.
• Unchanged: detection, Kestrel and adVNTR thresholds, each caller's own name, the decision profile and screening messages. The corpus is development evidence, not held-out validation.

2.0.44 (2026-10-08)¶

Report and log corrections¶

• Tier C under caller disagreement: when the two callers describe different events, the tier label reads callers disagree and its explanation says so. It no longer states that no coordinate could be computed. The genuine no-coordinate case keeps its wording.
• Column help: the adVNTR table's Position is described as a position within the adVNTR repeat unit, not within the Kestrel motif pair. The Kestrel Motif, Position and ALT help now say that a row is one edit of a resolved haplotype in a 120 bp pair frame, so the four fields alone do not reconstruct the allele sequence.
• Times in UTC: the render time of the sample and cohort reports is printed in UTC, as the run start already was.
• Embedded log paths: the copy of the pipeline log inside the report replaces the output directory with <output_dir> and the home directory with ~. pipeline.log on disk is unchanged.
• No false earlier-run warning: a fresh --extra-modules advntr run no longer warns that its own advntr/ directory came from an earlier run. A directory left by an earlier run still warns.
• Region-extract warning: an indexed input whose header declares several contigs, holds reads on one and has no unmapped reads now logs one warning that depth and variant support are lower bounds. No result file or exit code changes.
• Assembly detection logged once: the Assembly detected from chr1 length line is no longer written twice.
• Research pipeline decisions: No caller thresholds, result rows, tiers or screening messages changed.

2.0.43 (2026-10-03)¶

Release-independent golden checks and coverage ratchet¶

• Fitted-profile golden pin: Pins the fitted content without the generator version and derived profile ID. The installed version, production-derived identity, and full digest attestation remain checked. A version bump passes; a fitted-value change fails.
• Configured golden gate: make test-golden requires both external corpus roots. make check-all runs the golden tier when both are set and visibly reports when it cannot run it.
• Long temporary paths: The special-file reader test binds its Unix socket by a short relative name, preserving the test with long TMPDIR and worker paths.
• Coverage ratchet: Raises the branch-inclusive unit floor and independent advisory target to 91%, supported by 91.71% across Python 3.10–3.13. Patch coverage remains 80%; scripts-only coverage remains 88%.
• Research pipeline decisions: No caller thresholds, result rows, or report messages changed.

2.0.42 (2026-10-03)¶

Quality, coverage and length warnings recalibrated on real exomes¶

On 75 randomly drawn real exomes (GRCh38), 6.7% had a mean below the 100x threshold under 2.0.3, the only sample-quality warning that release had. Under 2.0.41, 24% carried a caution-toned grade or chip; with --estimate-vntr-length, 80% carried an amber mark of some kind, most of them the length badge. The thresholds below were re-derived from downsampling and simulation data, and the report wording was rewritten.

• Three depth levels instead of pass/fail, with lines per assembly: coverage_qc gains the value REDUCED for a mean between an assembly's low and adequate lines. It passes the gate. The new thresholds.mean_vntr_coverage_by_assembly sets 100x and 50x on GRCh38, and 290x and 145x on GRCh37. Downsampling 42 confirmed positive GRCh38 exomes (378 runs, repeated measures) detected the variant in 160 of 161 runs at 100x or more, 95 of 108 between 50x and 100x, and 48 of 109 below 50x. The GRCh37 lines are the GRCh38 lines times 2.89, a rounded policy translation: realigning the same reads from 214 exomes to both builds gave a GRCh37 window mean 2.89 times higher (5th–95th percentile 2.82–2.96) and the same Kestrel calls in 326 of 326 paired runs. Native GRCh37 alignments were not tested. Through 2.0.41 GRCh37 used 100x, about 35x on GRCh38, so more GRCh37 samples are now REDUCED or FAIL. A run whose assembly is unknown keeps the single 100x line, and so does a replaced config that sets only mean_vntr_coverage. The masthead chip is now VNTR depth with the words Adequate, Reduced, Insufficient and Not evaluated. See Coverage thresholds.
• Mean depth no longer qualifies a Kestrel call: a Kestrel call keeps its wording and its Finding grade at any mean depth. An adVNTR-only call is still graded Finding, with caveats at failing depth. Four of fourteen real positives at 69–87x had been graded down for depth alone. A call is still graded Finding, with caveats when the uncovered percentage exceeds its limit (50% by default) or coverage was not measured.
• Coverage is stated in sentences of its own: the new coverage_notes block of report_config.json holds four sentences, filled in with the measured figure and the line applied. A result without a finding gets one sentence at reduced depth, and a sentence plus a follow-up at failing depth. A region that is mostly uncovered is stated for every result, and no longer described as low depth.
• A below-threshold candidate no longer limits the grade: negative_subthreshold with passing depth is graded No finding. The message now says such candidates appear in 15–20% of real exomes without a Kestrel call (11 of 75 and 31 of 180 in two random Berlin samples; about one in five in a larger cohort count).
• One length tier, at 150: length_estimation.sensitivity.caution_threshold may be null and ships as null; 110 repeats was below the median real exome (53 of 67 estimates exceeded it; 13 of 67 exceeded 150). The note above 150 repeats on a report without a finding is reworded, prefixed Note: and no longer amber. A summary recorded with a numeric caution cutoff re-renders with its recorded tier.
• Report wording rewritten: each screening message is a result sentence, a flag sentence when the call is flagged, an adVNTR sentence, and for a call one sentence asking for confirmation by an independent method. No message states coverage, so the rule table has 26 rules instead of 50. Every sentence was shortened and made plain ("adVNTR was not run.", "The Kestrel call is flagged; see the Kestrel table."). Grade and result chips read from the new chip_labels block: Finding, corroborated, Finding, with caveats, No finding, inconclusive, No result, High precision, flagged, Negative, candidate below threshold; a token it does not name keeps the old spaced form.
• Mean icon and figure: the cross beside the mean marks a failing mean only. On a summary recorded before 2.0.8 the report shows and judges the corrected region-wide mean, like the verdict, and states the recorded covered-bases figure beside it.
• Integration compatibility: observation sets accept coverage_qc_overrides, which replace a case's expected coverage_qc token and nothing else. 2.0.42 uses one: the GRCh37 fixture example_7a61_hg19_subset_fast (185.92x) is now REDUCED.
• Custom templates: the shipped template reads the new coverage_qc_tone. coverage_qc_measured and screening_state.quality_metrics_pass are still passed and are deprecated until 3.0.0.
• Configuration validation: a per-assembly line that is not a finite, non-negative number is refused with a ValueError naming its key; a NaN line used to pass every sample.
• adVNTR precheck: its coverage summary is judged on the run's assembly, like the main one.
• Not changed: no caller decision or threshold. Kestrel and adVNTR result rows are unchanged. The coverage_qc value, the recorded length tier, policy and warning codes, and the digests derived from them do change.

2.0.41 (2026-09-27)¶

A sample with no Kestrel call no longer reads as a failed run¶

A cohort user asked whether their installation was broken. Every sample with no call had no IGV view, and a CRAM run logged four or five WARNINGs, none of which needed action. The missing view is expected: it is drawn around a Kestrel call, as in every release since the view was introduced. The messages and the report now say so.

• No-call samples log at INFO: BED file does not exist or not provided, No Kestrel data found in pipeline summary, fastp output file not found (BAM or CRAM input), and one of No insertion/deletion variants found or Final processed DataFrame is empty were WARNINGs on a sample with no call. Each is now an INFO line that names the cause. A FASTQ run that lost its fastp output still warns, and a summary with no Kestrel rows at all still warns.
• The IGV panel names its reason: The report used to say only that "this run produced no alignment session". It now distinguishes a sample with no Kestrel call (expected, and only when the run recorded Kestrel's no-call result) from three states that are not normal: a call whose region file is missing (naming that file), a leftover output.bed from an earlier run, and no readable Kestrel result. A --bed-file given to vntyper report is never treated as a leftover. The report panel and the log line are both worded from one decision (igv_absence.py), so they cannot disagree.
• Leftover region file: The Kestrel stage writes kestrel/output.bed only for a call, and nothing removed an old one. Re-running a sample that previously had a call into the same --output-dir, and getting no call this time, kept the old file, so the report drew an alignment view at a position the current run did not call. The stage now removes the file before it decides. A report re-rendered over such a directory ignores the file and says so.
• "Output directory ... is non-empty" on every run: vntyper pipeline opens <output-dir>/pipeline.log before the pipeline checks whether the directory is empty, so the check always found that file, and every run warned that prior results may be overwritten, even into a fresh directory. The run's own log (and a directory holding only it) no longer counts. The warning now names what the directory already held.
• Provenance: A report with no alignment view stated "igv.js … embedded in this file" in its Provenance section even though no library was written into it. It now states not included (this report has no alignment view).
• Docs: Troubleshooting explains when the view appears, and lists the log messages that are normal on a successful run. The FAQ no longer describes the view as read inspection for every sample. The SLURM example now exits with the sample's status, so a failed sample is recorded as a failed job.

2.0.40 (2026-09-24)¶

adVNTR cutoff axes in calibrate optimize (#269)¶

• adVNTR axes searched: --caller advntr is no longer refused. The new advntr_cutoff (call when p < cutoff) and advntr_min_support (call when read support >= the value) axes derive their breakpoints from native probe replays, and every candidate is replayed natively. The run stops unless the probe and candidate replays share their evidence, every tested candidate matches the legacy rule (replay_consistency), the adVNTR baseline reproduces the captured calls (baseline_parity.advntr), and the execution count is exact. --caller both searches both callers' axes, one axis at a time, on the either-caller union. Exact mode, background fitting and the rare-unit coverage guard are not searched. See Which caller is searched.
• Fold-local inventories for both callers: Each outer fold now derives its own breakpoints, endpoint sentinel and --max-breakpoints cap from its training samples (fold_admissibility: "training-derived-inventories"), so a held-out sample can no longer set its fold's sentinel or displace a capped breakpoint. Held-out Kestrel counts can change from v2.0.39 on the same cohort. Each axis states breakpoint_completeness.
• Research profiles run adVNTR: vntyper pipeline --research-decision-profile now runs adVNTR with the profile's derived legacy policy under fixed capture parameters, instead of failing. Exact mode is refused. A derived cutoff is tied to the adVNTR build recorded in the optimize report's provenance.
• Report key replaced: In the optimize report.json, search_scope.searched_caller (one string, which could not describe a --caller both search of both callers) is replaced by search_scope.searched_callers, a sorted list of the callers whose axes were searched. This is the only non-additive change to calibration-cutoff-report-v1; a consumer that read searched_caller must read searched_callers instead.

2.0.39 (2026-09-24)¶

Cutoff optimization and length-tier reporting corrections¶

• calibrate optimize --caller advntr is refused: No adVNTR cutoff axis is derived yet (#269), so every candidate replayed the same adVNTR policy and selection could only return the baseline. --caller both searches the Kestrel axes only. It holds the adVNTR arm at its baseline policy, records this under search_scope in report.json and states it on the page.
• Held-out estimate without leakage: Inside an outer fold, a candidate cutoff is admissible only if a training sample of that fold observed its value. Before this, a held-out sample's own feature value could create the cutoff its fold selected, which made the held-out estimate optimistic. Each fold records admissible_candidates.
• Reject-everything endpoint: Production floors pass a score equal to the floor, so the observed values alone never included the cutoff that rejects every row. Each derived axis now adds one endpoint sentinel just beyond the observed range (endpoint_sentinel).
• Truth-stratified folds: Cutoff outer folds are allocated stratified by truth label, so no fold is left without negatives when there are at least as many negatives as folds. Length evaluation is unchanged.
• Report page: The optimize report.html now shows the pooled held-out TP/FN/TN/FP, sensitivity and specificity with their intervals before any full-data number. It warns when a held-out rate falls below the requested floor and labels the full-data table as descriptive.
• Length typical error: The length estimate is no longer shown as "± 14". That value is the model's leave-one-out RMSE, not an interval: only 55 of 76 residuals fall within ±14, and the empirical 95% band is about ±29. The configuration field is now typical_error_repeats, and recorded uncertainty_repeats summaries still render.
• Docs: The sensitivity tier documentation now covers the limits of the tier evidence: cutoffs chosen on the same predictions, the high tier's 7/8 recall (Wilson 95% CI 53–98%), the selection-adjusted MAE, the asymmetric cross-cohort transfer (#336), and out-of-scope input.

2.0.38 (2026-09-23)¶

Kestrel failures are reported with their cause, not as "no usable VCF" (#338)¶

• Kestrel log contract: The pinned Kestrel 1.0.1 exits 0 after a fatal error. A missing or unreadable motif reference then writes no VCF, and a missing or empty k-mer count file writes a header-only VCF that would have read as a negative. VNtyper now reads each attempt's kestrel_kmer_<k>.log and treats any ERROR line as a failed attempt. The raised message quotes Kestrel's error and names the log. Across 32,545 existing Kestrel logs, the check flagged only the 3 runs that had genuinely failed.
• Input preflight: vntyper pipeline checks that both Kestrel JARs and both motif FASTAs exist before reading any input. If one is missing, it stops in under a second with the resolved path, the working directory it resolved against, and the fix, instead of failing every sample at the genotyping step.
• Terminal message: "Kestrel produced no usable VCF" now lists the per-attempt logs to read.
• Loud failures with help: Known Kestrel errors print a Fix: line, every new failure message links the Troubleshooting page, and a failed run now ends with one VNtyper failed: <cause>. ... Help: <link> line after the traceback.
• Docs: New Troubleshooting page. Every install-references example now installs into reference/, the directory the shipped config reads (several used ./references or /path/to/references, which reproduced this failure). The docs state that pipeline must run from the repository root, record that pipeline --reference-fasta exists from v2.0.10, and add a SLURM array example for large cohorts.

2.0.37 (2026-09-23)¶

VNTR length sensitivity tiers (#334, #335)¶

• Sensitivity tiers: A length estimate above 110 repeats records a caution tier and above 150 a high tier, with warning codes in length_estimation_warnings, for both the research and the approved length paths. Cutoffs and the displayed uncertainty (±14, the model's LOO RMSE) are configured under length_estimation.sensitivity and stay out of the resume identity.
• Report: The length card shows the estimate with its uncertainty and a tier badge. A caution-toned header notice appears only for the high tier on samples without a finding. Wording, the notice prefix and the cohort KPI labels are configured in the new length_sensitivity block of report_config.json; the report re-checks the recorded tier against the recorded estimate, policy and warning codes.
• Cohort: The statistics table and CSV/TSV/JSON exports add estimated_total_repeat_count (one decimal) and length_sensitivity_tier.
• Assembly detection: Contig detection falls back to the chr1 length, so subset BAMs are recognised. Header text is still reported as text, except that a header naming both hg19 and hg38 is resolved by the contig evidence.
• Docs: The packaged length model's count convention is documented as complete.

2.0.36 (2026-09-18)¶

MUC1 VNTR length estimation and cohort caller calibration (#269, #332, #333)¶

• Standard MUC1 VNTR Length Estimation: Added opt-in standard length estimation via --estimate-vntr-length and --length-model (#269). Features affine linear regression trained on sequence-verified PacBio long-read ground truth (\(N=76\) clean exomes across German and French cohorts, LOO-CV MAE 10.80, \(R^2 = 0.659\)).
• Reference Model Integration: Installed reference model grch38-standard-length-model-v1.json via vntyper install-references from vntyper-data release refs-v3, keeping python package wheel data lean while provisioning automated model installation.
• Cohort Calibration Optimization: Added vntyper calibrate cohort and vntyper calibrate optimize commands (#332), enabling empirical cutoff derivation and parameter fitting for Kestrel and adVNTR callers across cohort data.
• Custody Ledger & Exposure Safety: Hardened calibration target exposure tracking with atomic lock acquisition and defensive ledger preflight to prevent one-use confirmation races and data exposure leaks.
• Supplemental Reference Downloading: Added automated supplemental raw file installation to install_from_bundle, ensuring new common reference assets are downloaded and verified with provenance during bundle installations.

2.0.35 (2026-09-10)¶

adVNTR v2.3.0 upgrade and adapter read-through filtering (#331)¶

• adVNTR v2.3.0 upgrade: Pinned adVNTR to commit 6f9e5745f30ce5ffc3690ec843dc8f773b3d464f (v2.3.0 release).
• Opt-in v2.3.0 algorithms: Added CLI and configuration support for --filter-adapter-readthrough (filters reads with adapter read-through and short partial matches that cause false-positive indel calls) and --min-read-match-ratio (minimum fraction of read length required to be genuine match states in Viterbi alignment, default 0.60 when adapter filter is enabled).
• Version capability enforcement: Added version compatibility enforcement in advntr_options.py ensuring v2.3.0 options require adVNTR >= 2.3.0, and fail-fast refusal of -u/--update on adVNTR >= 2.3.0, while maintaining full backward compatibility with older versions.

2.0.34 (2026-09-10)¶

Representation-limited disposition and report cryptographic integrity anchoring (#313, #327, #330)¶

• Representation-Limited Disposition: Resolved issue #313 where Kestrel's VCF caller cannot represent deletion-insertion (delins) alleles. Emits an explicit representation-limited disposition with withheld positional names (e.g. frameshift +1, representation-limited), maintaining 100% frameshift carrier detection across all 400 simulated benchmark cases while preventing misleading positional names.
• Calibration & Golden Objective Invariants: Evaluated representation-limited outcomes separately from wrong names in calibration and golden evaluation, eliminating unjustified penalties against the calibration objective while preserving independent counting of positive control findings.
• BAM Refinement Safeguards: Safeguarded delins allele refinement in nomenclature_bam.py to preserve primary frameshift alleles and net lengths against conflicting BAM candidate events, preventing chimera construction and recording caller disagreement.
• Generation-Time Report Cryptographic Integrity: Resolved issue #327 by introducing multi-stage cryptographic integrity anchors in report_integrity.py. Normalizes line endings (\r\n and \r to \n) and intra-line tabs deterministically across platforms, computes SHA-256 digests across all pipeline decision files, and embeds an RFC 8785 canonical JSON hash of the pre-anchor summary to guarantee report authenticity and tamper evidence.
• Pipeline Stage Reordering: Reordered pipeline summary anchoring to precede summary report generation, ensuring run_id and verified integrity digests are directly injected into HTML reports and disk summaries.

2.0.33 (2026-09-08)¶

Integration contract update and identity dominance default enablement (#270, #295)¶

• Identity Dominance Default Enablement: Activated identity_dominance as the default Kestrel variant selection strategy across the pipeline and decision profiles.
• Integration Outcome Contract Update: Updated integration test assertions for example_6449 to reflect the biologically consensus candidate group metrics, enrolling versioned observation sets with exact append-only backward compatibility.

2.0.32 (2026-09-08)¶

Identity dominance selection, biophysical depth semantics, and junction boundary resolution (#270, #295)¶

• Identity Dominance Selection Engine: Resolved mis-named true positives where the correct candidate was present in pre-results but beaten under legacy coordinate-based sorting (#270). Groups passing candidate rows by canonical Molecular_Identity, ranks groups by an 8-tier lexicographic dominance hierarchy (confidence priority, unflagged status, canonical X-repeat context fidelity, physical peak alternate depth, assembly cardinality, peak depth score, peak haplo count, and min pos), and deterministically selects the optimal representative row with complete tie-breaking.
• Biophysical Read Depth Semantics: Replaced assembly-cardinality depth summation with physical peak alternate depth (max(Estimated_Depth_AlternateVariant) across motif assemblies supporting the winning identity), eliminating read double-counting across overlapping motif pairs while respecting Kestrel single-read multi-assembly alignment (#295).
• Junction Boundary Insertion Resolution: Resolved boundary insertions (such as dupA at repeat junction gap == 60) by identifying flanking sequence matching repeat unit 1 or 2 with minus-strand orientation priority, unlocking previously uncallable dupA frameshift mutations without sacrificing 100% false-positive specificity on normal controls.
• Configurable Kestrel Selection Strategy: Made selection strategy configurable via KestrelDecisionConfig(strategy=...) defaulting to "identity_dominance", while retaining "legacy" for full backward compatibility and historical reproducibility.

2.0.31 (2026-09-06)¶

Pipeline procfs descriptors, report integrity, and CI quality fixes (#328)¶

• Descriptor-Bound Procfs Output Fix: Preserved trailing /. on procfs descriptor paths in preflight_error_io.py and handle_pipeline, ensuring safe execution under containerized and restricted procfs mount environments (#328).
• Report Cryptographic Integrity & Tamper Anchors: Added pipeline summary anchoring and verification module (vntyper/scripts/report_integrity.py) computing deterministic decision digests and optional HMAC signatures.
• Anonymous Research Data Donations: Integrated client-stripped result donation endpoint (POST /donations/) with GDPR Art. 9 consent validation, integrity verification, and aggregation counters.
• Cumulative Usage Tracking: Added Redis HyperLogLog and cumulative job counting in worker tasks with retry deduplication guards.
• CI & Type Quality Enforcement: Resolved Ruff import sorting issues, Mypy type definitions across Celery tasks and Redis models, test marker hygiene, and integration compatibility observation enrollment.

2.0.30 (2026-09-06)¶

Web API v0.19.0, OpenAPI Docs Modernization & FastAPI 0.137+ Support¶

• Server-Configurable Analysis Modes: Added GET /options-config/ endpoint allowing frontend discovery of server defaults and forced analysis mode policies (default_advntr_mode, default_normal_mode, force_advntr_mode, force_normal_mode).
• OpenAPI Documentation & Swagger UI Theme Overhaul: Modernized Swagger UI with a WCAG AA-compliant theme featuring custom method badges, high-contrast parameter typography, and an aligned 1400px container layout.
• Accurate adVNTR Runtime Documentation: Updated documentation with real-data benchmark metrics (~30–90s for normal mode, ~1–3 min for adVNTR mode, with adVNTR genotyping completing in 3–11s on real alignment subsets).
• FastAPI 0.137+ Rate Limiter Compatibility: Patched fastapi-limiter for compatibility with _IncludedRouter route objects in FastAPI >= 0.137.
• Shared Brand Favicons: Added /api/favicon.svg and /api/favicon.ico endpoints in the web API matching frontend branding.
• Queue State CLS Elimination: Addressed navbar layout shift on queue updates using tabular numerals.

2.0.29 (2026-09-06)¶

adVNTR v2.2.0 integration and runtime optimizations (#326)¶

• adVNTR v2.2.0 upgrade: Pinned adVNTR to commit 8d7f4721c9e48b474bc9589cc35b7486ceace242 (v2.2.0 release), featuring a Cython-optimized Viterbi dynamic programming core (nogil) for parallel read decoding.
• Order-of-magnitude genotyping speedup: Per-sample genotyping runtime drops from ~9 minutes down to ~7 seconds at 8 threads, eliminating the operational requirement for BAM downsampling on standard coverage datasets.
• Opt-in v2.2.0 algorithms: Added CLI and configuration support for --prune-reverse (~25% speedup without sensitivity loss) and --rare-unit-coverage-guard (eliminating false positives in normal controls while preserving 100% frameshift sensitivity), exposed through --advntr-additional-commands and advntr_config.json.
• Pre-header comments and parser robustness: Enhanced adVNTR TSV/VCF parsing to handle pre-header #Input File: metadata without mangling read identifiers, and isolated the Context column.
• Modular options validation: Extracted advntr_options.py with allowlist validation, capability checking, and mutual dependency enforcement (--exact-frameshift-caller requiring --rare-unit-coverage-guard).

2.0.28 (2026-09-05)¶

Warning remediation and web stack refresh (#315)¶

• FastAPI lifespan and modern task management: Modernized the web service lifecycle management in docker/app using async lifespan context managers, eliminating legacy startup/shutdown handler warnings.
• Pandas clean semantics: Eliminated pandas downcasting warnings by replacing deprecated series transformations with explicit type conversion and .where() operations.
• Refreshed web dependencies: Pinned modernized dependencies across docker/requirements-web.txt and pyproject.toml with strict consistency contracts.

Test suite parallelization and performance optimization (#323)¶

• Parallel test execution with pytest-xdist: Integrated pytest-xdist into local Makefile targets (test-unit, test-fast, test-unit-cov) and the GitHub Actions CI matrix, reducing test run times by up to 5x.
• Targeted fixture and permutation optimizations: Optimized slow test suites (including bootstrap resampling and heavy test generation) without sacrificing branch coverage or mutation score.

Confidence grade (#324, Refs #173)¶

• Derived sample-level confidence grade: Added a derived, configuration-driven confidence_grade field to ScreeningSummary and configured confidence_grade_rules in report_config.json covering the ordered vocabulary (not-established, no-finding-limited, no-finding, finding-limited, finding, finding-corroborated).
• Masthead confidence grade chip: Rendered the computed confidence grade as a Confidence grade chip in the per-sample HTML report masthead via state_chips, with opt-in fallback preserving legacy report configuration compatibility.
• Render context contract declaration: Declared screening_state.confidence_grade in report_context_contract.py for custom-template API compatibility while keeping the template recursive parity audit intact.

Reporting floor split and decision profile revision 2 (#322, Refs #311)¶

• Split the MUC1 reporting floor (confidence_assignment.reporting_floor, 0.00469) from the lower edge of the Low_Precision band (confidence_assignment.depth_score_thresholds.low, 0.00469) and linked the GG depth-score threshold (alt_filtering.gg_depth_score_threshold) to the reporting floor.
• Closed a fail-open path in variant_parsing.py by requiring gg_depth_score_threshold in alt_filtering.
• Bumped decision profile revision to "2" and regenerated canonical profile and projection artifacts.
• Breaking change for custom decision profiles: Profiles generated under revision 1 fail closed with inventory fields differ because the schema validates exact inventory key equality; revision-1 profiles must be re-issued with vntyper calibrate.

Pipeline execution resume (#321, Refs #20)¶

• Checkpoint resumption with --resume. vntyper pipeline supports --resume to continue an interrupted run or re-render reports without repeating expensive computation.
• BAM/CRAM conversion, FASTQ alignment, Kestrel genotyping, and adVNTR genotyping steps are reused when their result files, checksums, and required sibling outputs are intact.
• Run-identity invariants (version, inputs, sample name, reference selection, and decision profile digest) are strictly verified before execution; any discrepancy causes a fatal refusal before running stages.
• Reused step records are carried forward into pipeline_summary.json with reused_from provenance recording the donor run's start timestamp.
• Default runs now emit a warning when --output-dir already contains files. --keep-intermediates remains an accepted compatibility flag.

Ordered confidence rule table (#319, Refs #173)¶

• Pure confidence decision layer: Refactored confidence_assignment.py to delegate confidence level assignment to a new pure module vntyper/scripts/confidence_rules.py.
• First-match rule table: Replaced six sequential, overwriting masked writes with an ordered, first-match rule table (ConfidenceRule, CONFIDENCE_RULES, assign_confidence_labels) implementing verified empirical semantics (rows 0 to 6) with the reporting floor as an outer precondition.
• Behaviour-preserving: All 54 cells in the confidence boundary matrix, mid-band demotion at Depth_Score == 0.00515, the #183 precedence of high tiers over active-region demotion, and the #184 fractional-depth gap fallback remain byte-identical.
• Mutation testing: Registered confidence_rules.py in scripts/mutation_test.py targets with pure unit tests pinning every row, the outer precondition, and rule ordering.

Cohort call frequency table and exports (#318, Refs #33)¶

• Cohort-wide call frequency and rare-allele flag. Added pure module cohort_frequency.py generating cohort-wide call frequency tables with 14 descriptive columns. Variant calls are grouped by Molecular_Identity (when status is unique or legacy-selected-among-multiple) or caller representation (Motifs, POS, REF, ALT). Denominator evaluates the full cohort roster (including samples without positive calls). Configurable cutoff via cohort.rare_allele_max_frequency in config.json (0.05) and CLI --rare-allele-max-frequency. Rendered as an interactive, fully escaped table in the cohort HTML report and exported as cohort_call_frequency.<csv|tsv|json>.

Atomic installation of BAM and BAI files (#320, Refs #314)¶

• Atomic subprocess writes. Every BAM and BAI file produced by samtools or bwa subprocesses is now written to a deterministic sibling .partial path and atomically moved into place with os.replace only upon verified completion. Interrupted or failing subprocesses no longer leave corrupt, truncated, or header-only BAM files under final names. In non-fast mode, the slice remains at .partial until merged, ensuring the public name never holds the uncompressed unmapped-free slice mid-run. Sliced BAM indexing is decoupled from the slice command and converges fast and normal modes. Destination validation contracts in alignment_target_io.py are widened to include all partial names.

SHARK provenance and read pairing (#317, Refs #312)¶

• SHARK stage records tool version, search parameters, and pairing verification. The command line explicitly specifies -k 17 -c 0.6 sourced from the runtime sidecar shark_config.json, matching SHARK 1.2.0's implicit defaults without modifying the empty /components/shark decision profile component.
• Tool version is probed via conda environment listing. Because SHARK 1.2.0 prints no version from --version or --help, vntyper.scripts.utils.get_tool_version probes mamba list -n shark_env shark --json and records <version>+<build> (e.g. 1.2.0+h077b44d_5), falling back to unknown without aborting if the probe is unavailable. shark is removed from UNPROBED_TOOLS.
• Step summary payload records provenance. pipeline_summary.json's parsed_result for SHARK Filtering gains shark_version, shark_k, and shark_c alongside the FASTQ paths and read counts.
• Asymmetric read retention fails closed. write_shark_step_summary verifies that kept_reads_r1 == kept_reads_r2 before building the payload, raising ValueError naming both counts and preventing incomplete step records on pairing divergence.

Flattened pipeline summary tables (#316, Refs #119)¶

• pipeline_summary.csv and .tsv carry the run's provenance and no longer embed JSON text. Every row now starts with run_* columns (schema version, decision policy, the six decision_profile_* fields, version, inputs, sample name, reference selection, resolved region, Kestrel counting mode and the adVNTR model); a single-row result explodes into parsed_result_data_<column> cells, any other row count is recorded as parsed_result_n_rows, and banner comments join into parsed_result_comments. A new pipeline_summary_rows.<csv|tsv> long table (step, row_index, field, value) is written beside each requested format and lists every result row, including the multi-row adVNTR and cross-match results. This is a user-visible change to the shape of two operator-facing files that no VNtyper code reads. --summary-formats still accepts unknown format names silently; validating them is out of scope here.

2.0.27 (Current)¶

Molecular identity, governed adVNTR evidence, and decision profiles¶

• Kestrel and adVNTR calls carry a canonical MUC1 molecular identity (#304, Refs #270). Pre-projection identity candidates are retained, callers are reconciled by molecular identity, and positive results publish the exact identity quartet through reports, cohorts, and exports. Negative schemas and the packaged legacy-selection-v1 selection are unchanged.
• adVNTR recurrent-state evidence is governed by a verified artifact (#305, Refs #267). Governed findings are retained while identity-insufficient observations are excluded from molecular agreement; the full evidence digest is snapshotted and verified across reports and cohorts.
• Every pipeline decision is owned by one complete, validated decision profile (#306, Refs #175). One packaged or explicit CLI profile is resolved before any artifact is written and threaded through the Kestrel, adVNTR, SHARK, nomenclature, reconciliation, report, and cohort paths; schema-3 profile provenance is snapshotted and verified, and hidden import-time decision ownership is removed.

Correction of the calibration replay, custody, and schema seams¶

• Calibration extraction derives its evidence from retained run artifacts under exact hash commitments, never from recomputation (#308, Refs #295). The candidate universe is bound to the verified profile snapshot rather than to profile_kind, so a generated profile with dominance disabled and an explicit custom profile with dominance enabled both replay correctly.
• Runtime dominance policy is installed and evaluated only when the resolved profile enables it, and the seam must report that it evaluated or the pipeline fails closed. Under an enabled profile the published kestrel_result.tsv now drops the same pre-result diagnostics the legacy path drops; the packaged profile never reaches that branch, so shipped output is unchanged.
• Negative adVNTR rows keep the exact legacy 18-column schema. Reconciliation skips a caller-negative surface, and cross-match reads the canonical VID == "Negative" row as negative evidence.
• Custody and one-use invariants are enforced with external bindings: a separately supplied custodian import bound to exact profile, protocol, partition, study, run-artifact, payload, and attestation bytes; a durable precommit; and flock-serialized consumption and retirement. The local verifier does not claim these checks prove organisational independence.
• The advisory mutation page covers all 17 registered targets: 1253 of 1486 mutants killed (84.3%), regenerated by the generator with a byte-identical render round trip.

Opt-in calibration and dominance evaluation¶

• Added four atomic vntyper calibrate operations for role-bound extraction, finite-grid safety-first fitting, fixed-profile validation, and one-use locked held-out evaluation. Generated profiles never auto-activate and cannot change shipped cutoffs or defaults.
• Added whole-locus record-dominance selection and closed abstention reasons. Haplotype records remain unweighted votes; XD remains optional minimum k-mer depth and can only veto a locus under a predeclared rule.
• Added static offline calibration reports, strict hashes, exact denominators, grouped bootstrap bounds, multiplicity correction, and local append-only custody receipts. Local receipts are not proof of external custody.
• The 200-mutated/200-control simulations reproduce the exact 154/136/18 shipped projection but remain previously examined development evidence. They are not external validation; Issue #295 stays open. Reporting an interval is not a clinical safety claim.

Kestrel BAM evidence semantics¶

• Phase 1 corrects the public ontology of Kestrel output.bam (Refs #295). Its alignments are resolved haplotype records, not sequencing reads; their counts are haplotype-record support. The optional XD tag is parsed separately as minimum k-mer depth and does not weight votes or alter names or tiers.
• Evidence flags now state their units. New Kestrel-specific thin-haplotype-record-support, low-haplotype-record-support, and low-kmer-path-support tokens are separate from genuine low-read-support; an undeclared source uses low-evidence-support. Current Kestrel BAM rows replace the pre-Phase-1 low-read-support token with BAM-specific thin-haplotype-record-support and low-haplotype-record-support tokens. Reports explain the same distinction; archived tokens retain their compatibility meaning.
• The internal BAM consensus interface uses truthful canonical fields. supporting_haplotype_records, fetched_haplotype_records, and distinct_edit_count replace the ambiguous constructor fields. Existing attribute readers retain read-only support, total, and n_distinct compatibility properties.
• Existing complete custom configurations remain compatible. The shipped thinness key is bam_thin_haplotype_record_support; the former bam_thin_support is accepted as a fallback. min_support_for_high_confidence, numeric values, record voting, candidate selection, allele names, ties, and tiers are unchanged.
• This is not the full #295 policy change. Molecular identity (#270), adVNTR artifact reliability (#267), threshold calibration and a calibrate command (#269), XD weighting, and new dominance or abstention rules remain out of scope for Phase 1. The later entries in this release address the in-repository parts, but independent externally custodied validation is still missing.

2.0.26¶

Report typography¶

• Gene symbols use nonbold italics. The MUC1 symbol in the report title no longer inherits the heading's bold weight.

Configured fastp quality boundaries¶

• Report cutoffs, displayed values, and pass/fail indicators now share the configured decision boundary. Exact JSON decimal spellings are retained for report decisions, rates use documented half-up rounding, and the public dictionary/float compatibility surface remains available (#290).
• Present fastp evidence is validated instead of silently treated as unavailable. Malformed values, impossible counts, and non-finite or fractional counts fail explicitly; a genuinely absent fastp artifact remains an unavailable optional report section.

Bounded comparator rules¶

• Flagging and cross-match rules no longer execute Python expressions. Kestrel and adVNTR flagging plus cross-match now share a data-only comparator DSL limited to the shipped comparisons and explicit bounded boolean composition (#286).
• Invalid rules fail before result publication. Complete rule trees, consumer columns, operand types, null behavior, and finite numeric values are validated up front; exact adapters preserve the documented shipped legacy expressions during migration.
• Genotyping behavior remained stable in the sealed parity cohort. The 1,600 baseline and candidate Kestrel/adVNTR jobs completed with zero artifact deltas, and the final combined head passed the Python 3.10–3.13 and Docker gates.

2.0.25¶

Versioned integration report observations¶

• Docker-fast report assertions now preserve historical truth while tracking current output. The 24 existing real-success compatibility contracts remain unchanged and ordered. Exact report-only corrections live in an append-only observation chain selected by the checkout's package version and bound to an ancestral provenance commit.
• Malformed or stale observation evidence fails closed. The compatibility gate rejects missing and unknown versions, reordered history, conflicting identities, non-report overrides, detached provenance, and JSON type changes such as an integer replaced by an equal-valued float.
• The two affected subthreshold Kestrel Docker cases use the shipped message without weakening their assertions. No production genotype or report-generation behavior changed; the complete Docker-fast tier and Python 3.10–3.13 unit matrix pass with the corrected oracle (#293).

2.0.24¶

⚠️ Required before deploying the web service¶

• Use a stopped, data-aware migration; this is not a rolling deployment. The ordinary-job handoff protocol now carries protected-spool identities and SHA-256 digests. Old and new API/worker processes are not rolling-compatible. Follow this sequence:
• Pause new submissions. Drain both the regular and long queues and all active jobs to completion; verify both queues and the active job count are zero. Never purge queued messages: purging can strand their protected-spool uploads.
• Stop the API, workers, and beat, then provision the named result_store.
• While services remain stopped and detached from the legacy host bind, either (recommended) copy existing unexpired output into result_store and retain a backup, noting that legacy bytes cannot be retroactively integrity-attested; or explicitly accept retirement and unavailability of those results and archive or remove the legacy store.
• Deploy the API, all workers, and beat; verify retained result access; then resume submissions.

Shipped Compose also moves outputs from its old host bind into the persistent, service-private result store mounted only into the API and worker. Together these changes protect against an actor with shared legacy input mount access. An operator override that exposes /opt/vntyper/output through a host/shared bind weakens this security boundary. Arbitrary same-UID code in either the API or worker service namespace is out of scope because it can access the private volumes or worker descriptors directly.

adVNTR concurrency and verified run inputs¶

• Version detection now remains authoritative under concurrent launches. The probe parses strict version answers from both stdout and stderr, tolerates unrelated libmamba lock diagnostics, and fails closed on missing, malformed, ambiguous, conflicting or incompatible output. Only a failed launch with the recognized transient lock error is retried; successful commands with invalid output are not.
• Only verified compatible successes are cached. The pipeline performs one early, run-scoped probe and freezes the exact command and tool paths used later. Failed, malformed and incompatible outcomes never poison the cache.
• adVNTR inputs and artifacts are bound to the verified run. Stale public outputs are invalidated through a fixed allowlist, active logs and model inputs are protected, and execution uses an atomic run-owned model snapshot with deterministic provenance.

Three synchronized 12-way launch rounds returned verified adVNTR 2.0.4 in all 36 attempts, including 33 launches with lock diagnostics. The final-SHA 400-sample cohort completed 400/400 jobs and produced 400 byte-identical intended result TSVs against the reviewed baseline, with zero schema, row or value deltas.

2.0.23¶

What the output asserts about a variant, beyond present or absent.

Two issues on the same surface. #266: a Kestrel candidate that failed only the depth gate was deleted, so the sample rendered identically to one where nothing was ever found. #267: the adVNTR Polymorphic_Call denylist labels calls as known artifacts on evidence that cannot carry the distinction, and seven of its thirty-two entries could never fire at all.

A suppressed candidate now leaves one line. kestrel_result.tsv gains a ## banner line naming how many events were below the reporting floor and how close the strongest came to it. Measured on the 400-run simulated benchmark, that lands on 22 of 22 false negatives and 1 of 200 true negatives — and on the false negative pair_3002 it reads 0.0040072 against a floor of 0.00469, where the true negative reads 0.00084203. That difference is what the output was throwing away.

It is a comment and never a row: parse_tsv routes # lines into comments, so the 10-column negative placeholder is untouched, cross_match and cohort mode never see it, and nothing that reads the table can mistake it for a call. The screening state becomes negative_subthreshold, declared a non-finding, so neither the masthead chip nor the summary box is styled as one. The reporting floor did not move, and a subthreshold candidate is still not a call (#147 stands).

Eligibility is strict: every structural gate explicitly True and the depth gate explicitly False. A row killed by flag_filter_pass carries a declared artifact flag (#174) and may have an excellent depth score; calling that "subthreshold" would say weak signal where the truth is strong signal, deliberately discarded. Set subthreshold_note.enabled to false to restore the previous output exactly.

Where adVNTR reports a finding on such a sample, the configured message now recommends orthogonal confirmation — @hassansaei's suggestion on #266.

The adVNTR denylist is 24 entries, not 32. Flagging runs after the pathogenic-frame filter, so an entry whose signed net indel change is not 1 (mod 3) is removed before add_flags is reached and can never fire. Seven were in that state and one was listed twice. Replaying both lists over the adVNTR output of all 400 simulated samples changes no Flag value.

advntr_calibration.json now records what existed only in a GitHub comment: the cohort (renome), the criterion, the fact that no observation counts were retained, and per entry whether it is confirmed or awaiting re-validation. New tests run the production filter over the live list, so a future dead entry fails the build rather than disabling itself in silence — as Poylmorhic_Call and RU == 7 both did.

One fix here is unrelated to either issue. The cohort-report fingerprint oracle (tests/unit/test_cohort_summary_oracle.py) excluded the embedded base64 PNG payloads — "a hash that turns red on pip install -U plotly stops being read" — but pinned their count. plotly 6.9.0 emits four of its own 114-character scaffolding images where 7.0.0 emits two, so the oracle went red on main itself the first time CI resolved dependencies afresh. Those two lines were the entire difference between the documents. The section is dropped and the fingerprint re-recorded; both dependency sets now produce a byte-identical document, and the property the section was for — an image with no payload is a broken chart — is now its own test that pins neither how many images there are nor that there are any. The charts' data is untouched and still pinned exactly.

Still open on #267: 23 of the 24 live entries await re-measurement against the re-analysed renome cohort. On current code, 8 of the 172 carriers adVNTR detects carry a Polymorphic_Call row — including every dupA carrier it detects. adVNTR records only the first inserted base and the repeat unit, so a recurrent benign insertion and a pathogenic duplication of the same shape are the same string; no denylist keyed on it can separate them.

2.0.22¶

The report carried the right facts in the wrong order.

2.0.21 made the per-sample report a faithful record: no code path can hide a variant row, and the document resolves nothing off-machine. What it did not settle is where any of it sits. Measured at 1440x1000 on a real specimen, the first variant table began at y=1580 of a 3142px document — half way down, 1.6 viewport-scrolls, behind 536px of provenance panels. The reconciled MUC1 name, its tier, its ambiguity interval, its repeat form and both callers' own names were columns 11 to 18 of a nineteen-column table, a third of which sat past the right edge of a container that gave no sign there was more. Nothing anywhere in the artefact said what a tier or a nomenclature flag meant.

<main class="container"> matched no rule in this template — .container was declared only in the cohort one — so the page had no gutter but the browser's 8px default and no measure at all: provenance lines ran 180 characters, the row-count line 224. At 390px the document scrolled 200px sideways. The headline, the one sentence stating what was found, tied "Pipeline Log" on size and lost to it on weight.

The masthead now answers "what is it called" beside "was anything found": the reconciled name, its tier, the ambiguity interval, the repeat form, both caller names and the flags, with the verdict prose held to a readable measure and the state chips — including both depth figures — in a column beside it. The tier badge says which tier-A condition the call missed, derived from the flags through the caller's own constants, instead of listing five possibilities and leaving the reader to choose.

The results tables lead the evidence and open on twelve essential columns, with the rest one radio away. It folds columns and never rows: the control names the full count, every folded column is either restated in the masthead or derived by the column beside it, and a reader with no script sees every column. Run provenance moved to the foot of the document; the BAM-header warning did not, because an alignment the pipeline cannot vouch for changes how every number below it reads.

The printed record is complete for the first time. A nineteen-column table lays out at 1442px on a 718px sheet, so 757px of it — the then-120 bp motif pair included — fell off every printed page with nothing saying so. The table now prints the columns that fit and every folded value prints beneath it, labelled, one block per row.

Table craft: pandas' border="1" grey is gone, data cells take the 13px step and real padding instead of the browser's 1px, numbers share a right-hand axis with tabular figures, the first column freezes, the scroll container is a focusable named region, every heading carries scope and its own explanation, and both tables have a caption. The alignment panel commits to a light ground in both palettes, because igv.js paints its own light theme and in dark mode rendered as a white slab with a 1.22:1 track label. Sorting is no longer offered on a single-row table, which took the document from 45 tab stops to seven.

Output-format changes¶

The presentation changes below are display only. Separately, the corrected Motif_sequence field in kestrel_result.tsv now contains the selected 60 bp repeat-unit half instead of the 120 bp pair record; no calling decision reads this annotation field.

• The adVNTR table's headings are English, matching the Kestrel table above it: NumberOfSupportingReads is Supporting Reads, Pvalue is P-value, RU is Repeat Unit, and the eight naming fields take the headings the Kestrel table already used. If you scrape the HTML, key on the new heading text.
• A semicolon-separated flag list is spaced, so a cell narrow enough to wrap breaks between flags instead of inside one.

2.0.21¶

adVNTR fetched a quarter of the GRCh38 MUC1 array; it now fetches all of it.

The read-fetch window is not configured anywhere — adVNTR derives it from the model's own content. get_length() sums the stored repeat units, and that sum was used as a genomic end coordinate. The two agree only when the units tile the array end to end exactly once, and the shipped hg38 model did not: it carried hg19's units verbatim, 840 bp for an array GRCh38 holds at 3,525 bp. Anything BWA placed in the remaining 76% was never fetched, on every GRCh38 run since the model was moved to that assembly.

Measured on 400 simulated samples (200 carriers / 200 matched non-carriers, ~208x, GRCh38): adVNTR detects 132/200 carriers before and 176/200 after, with the 68 samples that emitted no state at all reduced to 24. Cost is 2.6x adVNTR wall-clock and 1.3x peak memory — the HMM is unchanged, only the number of fetched reads grows. Four carriers are lost, all the canonical dupC at 12–22 supporting reads against 165–266x coverage, where the wider window's larger coverage denominator pushes them past p < 0.001. That figure is post-selection performance on the model-development cohort, not an independent validation estimate.

This requires adVNTR >= 2.0.4 and reference bundle refs-v2, and the pipeline now refuses to run rather than genotype with a mismatched pair: an adVNTR predating the recorded genomic end would silently reproduce the old window, and a v1 model carries it regardless of the binary. pipeline_summary.json gains a top-level advntr_model record — digest, schema version, VID, genomic interval, window and segment shape — so a result can be traced to the model that produced it.

Existing installations are unaffected until they reinstall references: bundles are pinned by release tag and digest, and refs-v1 remains published.

A rebuilt 53-segment model was also built and measured. It rescues all 24 carriers this release still misses but loses 15% of those it detects, projecting no better overall at 9.1x runtime, so it is not shipped (#272).

See #268, berntpopp/adVNTR#1 and berntpopp/vntyper-data#1.

2.0.20¶

MUC1 variants are reported by their literature name, with an explicit confidence tier.

Kestrel and adVNTR each report in their own internal coordinate frame — POS 67 G>GG, I22_2_G_LEN1 — and neither emitted the naming the MUC1 literature uses. Both are now translated into a single name such as 59dupC, carried in new columns (Nomenclature, Nomenclature_Tier, Nomenclature_Flags, Ambiguity_Interval, Repeat_Form, Nomenclature_Note, Nomenclature_Kestrel, Nomenclature_adVNTR) across the Kestrel and adVNTR result TSVs, the pipeline summary and the HTML report. Negative runs keep their existing narrower schema: no variant, no name.

Nomenclature is the reconciled verdict; Nomenclature_Kestrel and Nomenclature_adVNTR record what each caller said on its own, so a disagreement stays legible in either result file rather than being collapsed into a single opinion. adVNTR is optional — when it has not run its column is empty and the Kestrel result stands alone.

The name carries no c. prefix. c. asserts a coding-DNA reference sequence, and no transcript places this tract at positions 53–59. See MUC1 Nomenclature.

The tier decides what may be printed. Only tier A emits a bare number, and reaching it requires two independent callers agreeing after normalisation, a matching motif context and sufficient read support — no single caller can promote itself. Tier B states the event and its ambiguity window; tier C states the frameshift and stops. This exists because a confident wrong name is worse than an honest "allele undetermined": on the simulated benchmark, Kestrel places the whole insG family one position 3′ of truth, and those records look clean in isolation.

Where the VCF cannot express what the reads show — Kestrel has no representation for a delins — the existing output.bam is consulted for that call only, and adjacent non-matching CIGAR blocks are merged to recover the allele. The reads may supply a name the VCF lacked but may not veto one it has.

The one exception is corroboration by a second caller: where adVNTR and the reads independently name the same allele and Kestrel's VCF names another, the two outvote the one. "Independent" means a different caller — Kestrel's VCF agreeing with Kestrel's own alignment is one opinion, not two, and neither outvotes anything nor promotes a tier.

A read consensus may not settle a locus where the two callers describe different events: that is a conflict rather than a gap, and of the 20 benchmark loci where the reads were the only thing naming one, just 6 were right. Refusing the ten that rest on one or two reads gives up one correct name and withholds nine wrong ones.

Measured on 200 known-truth simulated samples: 134 named correctly, none of the tier-A names disagreeing with truth, and no name emitted for any of the 200 negative controls.

Also fixes a result row whose last column is empty being dropped from pipeline_summary.json as malformed, which could fail the cross-match step on a run that had otherwise succeeded.

2.0.19¶

Kestrel counts k-mers on more than one thread.

Changed¶

• Kestrel-mode k-mer counting runs as its own threaded step (#262). Kestrel 1.0.1 builds a KAnalyze CountModule and configures almost nothing on it: getCountModule() sets the k-mer size, the temporary directory, the post-count filter and the free-segment flag, but calls none of setKmerThreadCount, setSplitThreadCount or setThreads. Counting therefore ran at the compile-time defaults of one k-mer thread and one split thread, while being the large majority of Kestrel's work on an unmapped-dominated input.

VNtyper now runs the bundled kanalyze.jar as its own step and hands Kestrel the resulting indexed k-mer count. This is a supported Kestrel input, verified in the bytecode rather than assumed: IkcCountMap.preModuleRun adopts a supplied ikc file, sets rmLastTemp = false so Kestrel never deletes a count file it was handed, and skips its own count module.

The run's --threads budget is divided across KAnalyze's three concurrent stages rather than passed to each — they are independent stages, so passing --threads N to all three would start roughly 2.5N workers. Measured end to end on example_7a61_hg19_subset.bam: 29.5 s → 17.4 s at --threads 8 (1.68x). At the shipped default of --threads 4 the gain is modest (1.10x), because the budget is treated as a total rather than a per-stage multiplier; operators who want the larger win should raise --threads.

kestrel_settings.split_counting defaults to true. Setting it false restores the single-command path exactly — it is an operator kill switch, not a fallback: nothing selects it automatically, and a failed counting step raises rather than silently re-running the work internally. keep_ikc retains the count file for diagnosis, and the path a run took is recorded as kestrel_counting_mode in the pipeline summary.

• Unmapped-read extraction is one samtools process (#262). The whole-file extraction was a decode-to-SAM-text, pipe, and re-parse; it is now a single samtools view. Genuinely throwaway BAMs are written at BGZF level 0, and the produced alignment is indexed only when something will read the index.

• The startup tool-version log lists only the tools a run uses (#262). Keys such as shark and advntr no longer appear in an ordinary Kestrel run. The cohort, online and install-references stacks are imported on first use, taking import vntyper.cli from 0.35–0.39 s and ~116 MB to 0.19–0.20 s and ~82 MB.

None of this changes a genotype. The golden-cohort gate compares the two sides across 78 pipeline cases, three probes and the cohort cases: every genotype-bearing artefact is identical, as are the report tables, pipeline steps, CRAM read-set evidence and output.bed. The only artefact that differs is the executed command stream, which changes by construction; its delta was classified case by case rather than waived unread. See docs/development/golden-cohort-gate.md.

Fixed¶

• additional_settings can no longer desynchronise the two Kestrel commands (#262). When Kestrel is handed a pre-built count file it validates only the k-size, so a count file built with a different minimum count or minimizer size is silently accepted and silently different — which moves Depth_Score. Under split counting, kestrel_settings.additional_settings is now an allowlist of options that provably cannot reach the counter, and a count-affecting value such as --mincount 3 is refused with a message naming it. The shipped default is "", so no shipped configuration is affected; an operator who needs those options sets split_counting: false.

• A k-mer attempt never adopts an existing directory (#262). Each attempt owns <output>/kmer_<size>/, which is removed on every exit path. Adopting a directory that already exists would delete data the run did not write, so the run refuses to start instead.

Internal¶

• The golden-cohort gate gained an --expect-command-delta mode: a caller may declare in advance that the executed command stream changes on purpose, which keeps that one artefact out of the verdict while every other delta stays fatal. The rendered summary names each case whose delta was waived, because a declaration nobody reads attests nothing. The gate also compares output.bed, which the collector never read — #203's one-base interval shift would have been invisible to it.

• The waiver policy moved out of golden_cohort/compare.py into golden_cohort/waiver.py. It is the one rule in the harness that can turn a check off, and render_text had been recomputing half of it inline.

2.0.18¶

--threads reaches adVNTR, and adVNTR can use it.

Changed¶

• adVNTR genotyping is threaded (#215, #259). VNtyper now installs a hard fork of adVNTR, pinned at v2.0.3, in which the Viterbi decoder releases the GIL and the read-selection loop is parallelised. --threads reaches it: advntr_config.json sets "threads": null, which means inherit the pipeline's --threads, and an explicit integer still overrides.

Measured on example_7a61_hg19_subset.bam: 61.1 → 3.12 ms per decode attempt serially (19.6x), and end to end 203.3 s → 1.71 s at -t 16 (119x). Threading plateaus at 16 and regresses at 32, because roughly 6.2 MB of numpy allocation per call still runs under the GIL.

Before this, -t set only settings.CORES, which nothing on the genotype -fs short-read path read — so passing it changed the command without changing the runtime. That is why #215 was previously closed as resolved-by-investigation.

This changes no genotype. The decoder is byte-identical to the pristine upstream kernel over 2,000 stratified fixtures in two model contexts (4,000 decode-attempt rows), and seven of eight whole-BAM genotype -fs runs reproduce the pristine VCF and -aln sidecar exactly. Golden-cohort gate run 8 compares the two sides of this change across 78 pipeline cases, three probes and four cohorts: every genotype-bearing artefact is identical, and the only substantive command difference is -t 1 becoming -t 8 on the three adVNTR cases. See docs/development/golden-cohort-gate.md.

Fixed¶

• Extra adVNTR flags can no longer override an option VNtyper sets (#259). advntr_settings.additional_commands is appended after every option run_advntr builds, and adVNTR parses with argparse, where the last occurrence wins. A fragment repeating a managed option therefore replaced a validated value in silence — --threads 0 reached adVNTR even though the thread count is checked, and -o redirected the genotype artefact to a path pipeline.py does not read back. Every such spelling is now refused with a message naming the option and the setting that owns it, including abbreviations (--thr, -v), attached values (-t3) and single-dash flag groups (-pt3, -po).

• The thread count is validated where it is configured (#259). A non-integer, zero, negative or boolean threads now fails with a message naming the key and the file, instead of adVNTR's threads cannot be less than 1 from inside a subprocess.

Internal¶

• The golden-cohort gate accepts a module:attribute marker, so a branch that only modifies files can still prove each side ran its own tree. A branch that adds no module previously had no marker that distinguished the sides, and the harness refuses two sides that expected the same marker state.

2.0.17¶

Say which coverage number can be compared, and which cannot.

Added¶

• Coverage figures that can be compared between assemblies (#222). Mean coverage over the VNTR region is about 2.7x higher on GRCh37 than GRCh38 for the same sample -- and about 4.3x over the repeat array alone -- because the two assemblies represent different amounts of the repeat array -- roughly 13.5 60-bp units against roughly 58 -- and the same reads pile onto whichever copies exist. Nothing in the output said so, and harmonising the configured windows would not have fixed it: the difference is in the reference, not the window.

coverage_summary.tsv gains vntr_flank_mean_depth, which measures unique flanking sequence and is therefore a property of the sample rather than of the assembly, alongside vntr_array_depth_sum, vntr_array_depth_sum_per_unit_length and the geometry and counting policy each was measured under. The array figures are comparable between builds only under the policy named in depth_counting_policy -- they agree to within 1% under samtools depth defaults and diverge to a ratio of 0.31-0.55 under MAPQ >= 1 -- and that token is emitted so the number cannot be read as a general property.

No existing value changes. The eight statistics, the QC gate and genotyping are untouched. A run whose region is not the configured VNTR window records the new columns as NA rather than 0, because zero would claim the region was examined and found empty.

Fixed¶

• A malformed coverage column no longer discards the ones after it. parse_coverage_stats coerced every column in one block and returned on the first failure; coverage_qc is the last column, so a single bad number took the QC verdict with it. The original columns keep their previous behaviour exactly; the columns added above are isolated so neither can cost a reader the other.

2.0.16¶

Pin what we build on, and let no attempt inherit another's leftovers.

Added¶

• vntyper install-references --derive-only rebuilds the derived reference files from genomes and seeds already on disk, downloading nothing. Recovering a tree missing one of them previously meant a full --from-source run — re-downloading and BWA-indexing six chromosome FASTAs to rebuild three small ones — so in practice it was done by hand, with samtools faidx retyped from the config, and by hand is where an unverified reference file comes from. Every output is checked against its committed digest. A derivation this tree cannot rebuild is skipped rather than failed, and whatever is already at its path is verified instead, so all three are checked either way.

Fixed¶

• A discarded Kestrel attempt no longer leaves its SAM behind for a later one to convert (#255). Every k-mer size writes to the same output.vcf and the same output.sam. A discarded attempt removed the VCF but left the SAM, and a later successful attempt converts whatever occupies that path into output.bam — the alignment the report's IGV track shows. The genotype comes from the VCF and is unaffected, which is why it was easy to miss.
• convert_sam_to_bam_and_index checks samtools' exit status instead of inferring it (#255). Success was inferred from os.path.exists on the outputs, which conflates "samtools succeeded" with "a file is present". A failed view still leaves a truncated BAM, so the SAM was deleted and the truncated BAM kept. Both results are checked now, and the SAM survives a failure because it is the only copy of the data.
• A reference entry pointing at a 404 is gone (#253). It was unreachable — the installer drops any raw_files entry that is also a derivation output — so the issue's claim that --from-source is broken by it is false and is corrected on the issue. A 404 was the mildest way it could have failed: a URL resolving to a stale but live file would have installed silently wrong reference data. Tests pin the rule rather than the entry.
• The unauthenticated download window is 3 days rather than 7 (#189 — a mitigation). /download/{job_id}/ takes no credential, so the retention window is the exposure window: the link is the capability. This reduces exposure duration, not exposure. The completion email now names the window and says the link carries no password.
• .env.example's retention knobs are now actually configurable. MAX_RESULT_AGE_DAYS and COHORT_RETENTION_DAYS were presented as operator settings, but compose passed a literal for the first and never passed the second, so uncommenting either configured nothing. Default behaviour is unchanged; the documentation becomes true.

Changed¶

• adVNTR is pinned to a commit, not a branch (#254). advntr_genotyping.py cites exact upstream line numbers as evidence for why adVNTR runs at -t 1, and a branch can move under those citations. GIT_COMMIT now names 05fd98a4, and the installer checks it out and aborts if it is absent rather than silently building the branch tip.
• install_advntr.sh resolves its config beside the script, not in the working directory, and resolves symlinks to get there — the image runs it from elsewhere, so a bare relative name was never sourced and the pin had no effect on the image it exists to pin. -c/--config now replaces the shipped config rather than layering over it.

2.0.15¶

Stop claiming more than we know. Five defects in which VNtyper stated something it had not established.

Fixed¶

• A Kestrel VCF that cannot be parsed no longer produces a Negative (#223). A run that exited 0 having lost its #CHROM header discarded every indel record it carried and reported a confident negative genotype, with no error logged at all. The zero-byte case is the mild one; the severe one converts a positive into a negative. Each configured k-mer size is now tried, and a run where none produces a usable VCF fails loudly. A valid header with no records is still a legitimate negative and is unchanged.
• summarise_coverage refuses a depth table longer than its region (#224). It previously reported percentages above 100 -- four uncovered positions in a two-base region gave percent_uncovered: 200.0. Region coordinates are also validated before samtools depth runs, so a reversed or oversized region fails immediately instead of writing an unbounded depth file; a reversed region used to reach the report as a negative length beside a passing coverage verdict.
• The adVNTR thread count and output format come from configuration alone (#247). Both code fallbacks contradicted the shipped advntr_config.json, so dropping either key changed the emitted command silently.

Changed¶

• The golden-cohort harness option --advntr-threads is now --advntr-case-threads, and the side.json key with it (#247). The value is unchanged. The old name read as the thread count adVNTR received, which was never true: adVNTR runs at -t 1 from its own configuration and never sees the CLI value.
• docs/pipeline/scoring-and-confidence.md describes what a confidence tier says about the evidence rather than what a reader should do with it (#250). VNtyper is research use only, which that page previously contradicted five times.
• --pseudonymize-samples and the cohort documentation state what the pseudonym does not protect against (#227): the digest is unsalted and unkeyed, so a shared report must be treated as identifying. Documentation only; the digest is unchanged.
• Planning documents are no longer tracked. docs/ is now strictly the published site.

2.0.14¶

CRAM runs fail under Docker with Too many levels of symbolic links (#238).

Every Docker user whose CRAM reference has no .fai beside it is affected on 2.0.10 through 2.0.13, including ghcr.io/hassansaei/vntyper:latest.

CRAM preflight retained the reference index htslib generates by opening it and then replacing that same pathname with a symlink to /proc/<pid>/fd/<n> — a descriptor whose own recorded path is the pathname just replaced. The entry pointed at itself:

reference.fa.fai -> /proc/22/fd/7
/proc/22/fd/7    -> …/reference.fa.fai (deleted)

Linux tolerates that because it jumps straight to the descriptor's directory entry; a container runtime that resolves procfd links by re-walking the link text closes the cycle and every consumer that opens the index gets ELOOP. It is the only binding whose source and destination are the same path, which is why a reference with a shipped .fai decoded fine in the same run. The entry is now retained under its own name rather than replaced, and both binding classes additionally prove an installed run-local view can be opened through its published pathname before handing it to samtools.

Input and output must be two different host directories.

The same report exposed a second defect. Three containment guards compared pathnames only and never os.path.samefile, so one host directory mounted at two container paths — -v .:/opt/vntyper/input -v .:/opt/vntyper/output — defeated all three: the run wrote its output tree inside the directory holding the patient alignment, operator BEDs and references inside the output root went unprotected, and the log-path guard, which runs before the log directory is created, left a directory in the patient tree even when a later guard refused the run. All three now compare by inode and name both pathnames in the rejection.

This is a bypass, not a new policy: the identical layout with a single mount was already rejected. Runs using the documented separate-directory layout are unaffected. README.md and the Docker user guide now state the requirement and show both forms.

2.0.13¶

One reference resolver, and reference data from verified bundles (#217, #163, #152, #193).

Three genotype-affecting bugs are fixed. --reference-assembly hg38_ncbi and hg38_ensembl silently loaded the GRCh37 adVNTR database, because an incomplete map defaulted every unrecognised label to hg19. SHARK filtered against the hg19 MUC1 region regardless of --reference-assembly (#152): 40.6% of the hg38 region's canonical 17-mers are absent from the hg19 region, and the correct reference retains 3.2-34.7% more reads across seven cohort samples. And after install-references --output-dir, config.json kept its shipped relative paths, so the run died (#163) - the installer wrote keys nothing reads while the seven keys the pipeline does read were never updated.

Eight assembly labels now resolve through one registry to six physical genome files, plus two adVNTR databases and two MUC1 region FASTAs keyed by coordinate system. All five readers and the one writer go through it, so the writer and the readers cannot disagree again.

Reference data moved out of this repository into versioned, checksummed release bundles published from berntpopp/vntyper-data. install-references downloads each asset, verifies it against the digest committed here, extracts it with a hardened extractor and activates it atomically; a per-file provenance ledger records what was verified, and only files with a record are written into config.json. --from-source still rebuilds everything from upstream.

A fresh clone has no reference data and must run vntyper install-references. The Docker image installs the bundle at build time and now carries all six physical references, so every assembly label works out of the box; it is correspondingly larger. A configured-but-missing reference now fails immediately, naming the assembly, the key, the missing path and the command that fixes it.

2.0.12¶

Valid mixed alignment conversions are now routed losslessly (#233). When R1 and R2 contain equal non-zero record counts, every non-empty R1, R2, other, and singleton FASTQ is passed exactly once, in that order, to one Kestrel 1.0.1 sample. The files are not concatenated or recompressed. Unequal or one-sided mate outputs still fail before Kestrel as invalid. This restores BAM, CRAM, and adVNTR runs rejected by 2.0.10 and 2.0.11 while retaining genuine singleton and unpaired reads instead of silently discarding them.

2.0.11¶

When both --keep-intermediates and --delete-intermediates are supplied, --delete-intermediates now wins as the CLI help has always documented. Earlier versions kept the intermediate BAM in this conflicting form.

The integration test matrix covers more outcome and input states. Successful cases now assert both present and absent intermediate/archive artifacts and run an independent paired b178 FASTQ case without SHARK (#71). Issue #226 closes from pre-existing reference-dependent CRAM fixture implementation and tests; 2.0.11 adds no duplicate fixture code.

Runtime quality gates now measure the root scripts. make type-check includes scripts/ in mypy (#204), and all 35 Python files there are part of branch-inclusive coverage: the final measurements were 93.94% scripts-only and 89.17% combined across 5,072 unit tests (#211).

Mutation sweeps are crash-safe for production source. Each mutant runs in a disposable detached worktree over the current non-ignored working state, with import provenance and a known-killed canary checked before atomic reports are installed in the real checkout. Even an unhandled interruption can leave only the disposable worktree, not a live mutant in production source (#208).

Broad-exception behavior is measured rather than globally rewritten. The reviewed BLE001 inventory is 103 diagnostics normally and 108 including explicit suppressions; fail-open handlers are classified by symbol and linked to behavior tests. BLE001 remains outside Ruff's global selection, with no global fallback behavior rewrite (#219).

Release automation is exact-commit and recoverable. The controller verifies exact-SHA CI and Docker evidence, promotes the tested GHCR digest to semantic-version and latest aliases with idempotent retry/recovery, and separates unprivileged package building from PyPI Trusted Publishing through OIDC (#214, #218). Publication and stable aliases remain pending the first gated release.

Presentation uses the generation name VNtyper 2. Nine targeted VNtyper 2.0 presentation labels now say VNtyper 2, while semantic versions, historical statements, identifiers and machine-readable names remain unchanged (#220).

2.0.10¶

Milestone 4, "CRAM and input robustness": closes #213, #225, #209, #178, #165 and

161.¶

A converted alignment that contains more than one read layout now stops instead of discarding one layout. In the measured golden cohort, 32 of 50 base BAM cases failed rather than discard the non-empty single FASTQ produced beside their paired FASTQs. VNtyper 2.0.9 silently ignored those reads; 2.0.10 names every produced FASTQ and record count and refuses the mixed layout. This no-read-drop rule is a hard invariant, not a configuration option. Regenerate the alignment with consistent pairedness and mate flags, or provide a homogeneous paired-end or single-end FASTQ input; do not suppress the error by discarding the named reads. Pure single-end BAMs and --fastq1 without --fastq2 are now supported through fastp, BWA and Kestrel.

Superseded on 2026-08-11 by #233. The paragraph above accurately records the policy shipped in 2.0.10. Current behavior retains the same no-read-drop invariant but routes equal R1/R2 plus singleton/other reads losslessly; only unequal or one-sided mates fail as invalid.

BAM and CRAM inputs are preflighted through a run-local alignment view. Missing indexes are built beside that view, where htslib can resolve them, without writing into a read-only patient input tree. The alignment and freshly generated index are opened and retained for the plan lifetime; post-preflight random-access commands use that exact index inode, so replacing the public BAI/CRAI name cannot retarget a slice or depth calculation. Existing BAI, CSI and CRAI spellings are protected by format, and every derived BAM/FASTQ/log destination is checked before a processing stage starts.

For BAM or CRAM input, the output root must stay outside the directory containing the alignment and all of that directory's descendants. A layout such as sample.bam beside results/ is therefore rejected: results/ is still inside the protected patient input tree. Put alignments and results under separate roots, for example --bam inputs/sample.bam --output-dir results/sample/, or choose an output path elsewhere.

Reference-dependent CRAM fails before processing unless one candidate can decode the real -P slice shape. --reference-fasta, configured CRAM/BWA references and local htslib resolution are tried in configured order; coverage receives the same selected reference. Ambient network refget is disabled by default, REF_PATH is restored on every exit, overlapping in-process CRAM scopes are serialized, and web jobs transport a curated preflight code/message without exposing worker paths. Header-derived local references are confined to the CRAM's own directory; use --reference-fasta or a configured reference for an external path.

Unmapped CRAM extraction is fail-closed. auto uses indexed '*' extraction only when idxstats proves no placed-unmapped reads and the exact literal-'*' flag-4 count equals the terminal idxstats count; malformed or unequal evidence selects the stream path, and forcing an unsafe indexed path raises instead of losing reads. On a zero-placed 7a61 CRAM, whole-file decoding and idxstats each reported 622,690 records while the raw indexed fetch returned only 2,690; the count proof refused that 620,000-read loss.

Archives and web delivery are ownership-bound. Result archives are built and served through descriptor-anchored paths, reject unsafe source aliases, and are quarantined or revoked when a later worker status step fails. Cohort inputs are snapshotted through held descriptors, so replacing a public path cannot change the bytes a child consumes.

Chromosome naming now votes only across configured primary-contig patterns, requires a strict majority and returns unknown for ties or zero classified contigs. Threshold and pattern configuration is validated rather than accepted partially.

On the 18 golden-cohort BAM cases that both 2.0.9 and the measured milestone candidate can complete without discarding reads, three alternating runs measured medians of 90.03 s and 87.52 s, respectively. The non-overlapping ranges (90.00–91.13 vs 87.49–87.83 s) record a 2.8% median improvement under the predeclared rule. The harness recorded candidate 388f157; its published-history runtime-tree equivalent is 470fdd6. Later CRAM-reference and single-end fixture changes are outside this measured BAM arm. Subsequent alignment-binding identity hardening adds only a fixed, tiny number of metadata checks and is timing-neutral by inspection; archive identity hardening runs only under --archive-results, which the timing harness did not enable. This does not relabel the measured hash as final HEAD.

2.0.9¶

Milestone 2, "Web service and cohort integrity": closes #216, #207, #206, #205, #201,

167 and #162, plus #210 from the CRAM milestone because #162's acceptance criterion¶

depends on it.

Cohort pseudonyms change format, and existing pseudonymization tables do not carry over. A pseudonym was the prefix plus five hex characters of MD5 -- 20 bits, which collides with probability ~37.9% at 1,000 samples, and sample_mapping is keyed on it, so a collision silently merged two patients into one row and sample_categories counted them once. It is now 12 hex characters of SHA-256 (~1.8e-9 at 1,000 samples), with the algorithm and length in config.json under cohort.pseudonym. Re-run any cohort whose table you need to match new output.

Zip cohort inputs report a different sample name. A zip whose pipeline_summary.json sits at its root -- the layout the web worker produces -- took its reported sample name and its sort position from tempfile.mkdtemp, so both differed on every run. The identity now comes from the input file the run itself recorded, and the ordering key contains no temporary component, so two runs of one cohort produce byte-identical CSV, TSV and JSON exports. Directory inputs keep the order they had.

Two samples that cannot be told apart are now reported separately or refused, never merged. Samples sharing a name are reported under <input>/<name>; two inputs that share a name as well abort with a message naming both. A cohort reached through two spellings of the same path -- an absolute parent and a relative child -- is one sample again rather than two.

A read-only input directory works. Nothing in the vntyper package writes beside the input any more: the samtools quickcheck log goes to the run's output directory, and a BAM index is resolved in htslib's order and built into the output directory when absent. On the web service this also makes the worker's input-directory cleanup reachable for the first time -- every job used to leave one file and one directory behind, permanently, on the volume every job shares.

Two XSS fixes in the reports. The flag tooltip decoded the server's escaping and reparsed it, so a Flag value from a supplied pipeline_summary.json could execute; both templates now build that markup through DOM APIs. The two igv-reports fragments were interpolated into a <script> block behind a non-empty check and are now re-serialised through json.dumps with every < escaped.

vntyper report produces the IGV panel again. It passed no VCF at all, and resolved the BAM and BED only under --input-dir; all three now resolve from one effective run directory, and --vcf-file was added.

No genotype field changed: the golden-cohort gate was run over the full 60-case matrix against 2.0.8 and every Confidence, Flag, POS, REF/ALT, Motif*, Depth_Score and VID is byte-identical.

2.0.8¶

Milestone 2, "Correctness of reported numbers": closes #171, #172, #174, #203 and #212.

Coverage numbers change for every sample with an uncovered VNTR base. Historical reports are not directly comparable with 2.0.8 output. For any sample whose region span is unchanged, the corrected mean is recoverable in closed form from the old output:

mean_corrected == cov_mean_old * (1 - cov_percent_uncovered_old / 100)

Reported-output changes¶

• Every VNTR coverage statistic is now computed over the region, not over the covered positions (#171). samtools depth was called without -a, so it emitted only positions with at least one read, and mean, median, stdev and min divided by the covered-base count. The bias was largest exactly where coverage is patchy: across 8215 sample rows from seven cohorts, 1585 carried an inflated mean and 61 were reported as meeting the 100x QC threshold while falling below it. min is now 0 wherever any base is uncovered, which it always should have been. percent_uncovered keeps its value but changes derivation — it counts zero-depth positions rather than subtracting the row count, because with -a that subtraction is always zero.
• The quality gate uses both coverage metrics (#172). percent_vntr_uncovered was configured with a threshold of 50.0 and compared to nothing; it drove a report icon and no decision, so a sample with acceptable mean coverage and half the VNTR uncovered passed QC. That is the wrong failure mode — a patchy VNTR is where a frameshift call is missed. quality_metrics_pass now requires both. The verdict is emitted as an explicit coverage_qc column in the coverage summary, in pipeline_summary.json, in the report, and in a new cohort_stats.{csv,tsv,json} cohort export. docs/pipeline/reports.md already documented this threshold as enforced; it was not, and now is.
• Known 4 bp insertion artifacts are excluded from calls (#174). A row flagged False_Positive_4bp_Insertion reached kestrel_result.tsv and mapped to High_Precision_flagged, which the report treats as a positive finding — a known technical artifact presented as a positive MUC1 call. A new flag_filter_pass gate excludes rows carrying a flag declared in kestrel_config.json's new artifact_flags list. Advisory flags such as Low_Depth_Conserved_Motifs are unaffected and still only deprioritise. The excluded row remains in kestrel_pre_result.tsv with the gate False, so the evidence is never destroyed, and emptying artifact_flags restores the previous behaviour with no code change.

Fixes with no effect on genotypes¶

• The POS_fasta rebase that wrote to a discarded column is removed (#203). motif_correction_and_annotation subtracted position_threshold from POS and, because Series.mask keeps its name, DataFrame.update wrote the result back to POS — a column nothing reads afterwards. The rebase was also wrong: Motif_fasta is a verbatim copy of the VCF #CHROM, which names a 120 bp pair record of All_Pairwise_and_Self_Merged_MUC1_motifs_filtered.fa, so POS is already a coordinate in that record's space. POS_fasta values are unchanged.
• output.bed writes 0-based half-open intervals (#203, rider). It emitted [POS, POS+1), naming the base after the variant, so IGV highlighted the wrong position on every row. generate_bed_file had no test of its output content; it does now.

Safety¶

• The Kestrel stage is never skipped because output.vcf exists (#212). The skip was unconditional, undocumented, untested and gated on no flag, and because it returned it also skipped the post-processing that produces kestrel_result.tsv. Re-running into a directory left by an interrupted run — after a crash, an OOM kill or a cancelled job — could therefore re-report a stale result, or produce no result file at all, which both the report and cohort mode render as a negative. A stale VCF is now removed with a warning and Kestrel runs. If no k-mer size produces a result, the run raises instead of returning silently.
• A step whose result file is missing is recorded loudly (#212). record_step logged nothing and stored an empty data list, indistinguishable from a step that legitimately found nothing. It now logs an ERROR and sets result_file_missing. The key is added only when the file is absent, so a normal run's pipeline_summary.json is byte-identical.

Not included¶

#171's region harmonisation. The hg19 span is 1501 bp and the hg38 span is 4501 bp, so mean coverage is not comparable across builds. Choosing one homologous span and a flank convention is a domain decision that restates every hg38 number ever reported, and it is tracked separately so that each gate diff stays attributable to one change.

Coverage¶

The floor moves 80 → 81 (branch-inclusive, 81.70% measured) and the target with it. coverage_qc.py and coverage_stats.py are both at 100%; patch coverage of this branch is 100%.

2.0.7¶

Follow-ups to #179: closes #181–#188, #192 and #194–#197, fixes the defects found while doing so, commits the golden-cohort gate for the first time, and raises the coverage floor from 70 to 80 on a strictly harder measurement.

Branch coverage is now on. branch = true means an if that is entered but never taken no longer counts as covered — the failure mode this codebase actually has. The floor moved with each measurement (70 → 74 → 79 → 80) and was never lowered to admit a run. Statement-only coverage of the same suite is 80.77%, so the two numbers are not interchangeable: deleting branch = true would raise the reported total while covering strictly less.

Reported-output changes¶

Every one of these is a fix to a rule that was already meant to hold. They are listed separately because each can change what a report says.

• adVNTR compound states are judged on the signed net indel, not its magnitude (#182). abs(Insertion_len - Deletion_length) discarded the direction of the change, and a mixed state satisfies both presence guards, so either arm could admit it — I9_2_A_LEN3&D50_2 (Δ = +2, the non-pathogenic frame) was reported through the deletion arm. Both arms now test the sign, which makes the pair equivalent to the single signed rule Δ ≡ +1 (mod 3) that Kestrel already applied.
• A compound state's inserted length is the sum over its LEN tokens (#192). I9_2_A_LEN9&I50_2_A_LEN3 is 12. The previous parser coerced any multi-part remainder to zero, so a pure-insertion compound collapsed to a net change of 0 and was dropped in silence.
• The Depth_Score mid-band is a closed interval (#184). A score of exactly high_threshold is Low_Precision, not High_Precision.
• A malformed motif ID drops its own row, not the sample (#179 W8). The check was an aggregate .max() over the whole column, so one row with two dashes suppressed every call in the sample — a Negative report and exit 0 — while a row with no dash rode through unnoticed whenever a sibling row had exactly one.
• Potential_Duplicate sorts on Depth_Score alone, stably (#197).
• The adVNTR "positive flagged" report rule is guarded on VID (#188/D10), matching the "positive" rule beside it; it could previously fire on a Negative call.
• Indels are classified by length difference (D2), so a multi-base-REF insertion is no longer routed into the deletion file.

Fail-closed, where a silent wrong answer was possible¶

Each of these used to log and continue. A run that would previously have produced a quietly wrong report now aborts.

• cross_match raises when a match_logic rule cannot be evaluated — "no match" was indistinguishable from genuine discordance.
• filter_final_dataframe raises when a required gate column is missing from a non-empty frame; a missing gate is not a permit (#185).
• confidence_assignment requires its calibration keys instead of defaulting them. The old defaults were not neutral: high fell back to 0.4 where the shipped config sets 0.00515, a factor of 78.
• filter_vcf asserts the pinned Kestrel 1.0.1 allele contract, so a jar swap is loud.
• extract_unmapped_from_offset raises on a truncated or non-BAI index instead of scanning from offset 0.
• validate_bam_file honours its documented ValueError — the branch that raises it was unreachable.
• motif_processing refuses use_uniform_filtering with an empty GG allowlist, which would delete every GG alternate including canonical dupC (#186). Cannot fire on the shipped config.

Web service and CRAM¶

• An accepted CRAM is now run as a CRAM (#188). The endpoint has always accepted .cram, but the worker hardcoded --bam, and the index fallback named a .bai that is never written beside a CRAM — so the existence check never found the index the worker had just built, and cleanup left the real .crai on the shared volume.
• Patient files come off the shared volume first. The cleanup block used to open with two unguarded Redis calls, so a broker that became unreachable as the task exited left the alignment and its index behind — and replaced the pipeline's own exception with a connection error.
• CRAM unmapped-read extraction waits for its writer. The tee >(...) process substitution let the shell return while samtools was still flushing: measured on a 600k-read CRAM, samtools merge ran against 199,797 of 200,000 reads and accepted the short file with a warning.
• Alignment extensions are matched case-insensitively across all three layers, so the SAMPLE.CRAM the upload allowlist accepts is no longer rejected by the validator. FASTQ stays case-sensitive on purpose — fastp reads reads.FASTQ.GZ as text and reports 0 reads with exit status 0.

Evidence¶

Gated by golden-cohort run 5 (4fd638a → 9816f86), and no pipeline source changed after the gated commit. The harness itself is committed for the first time, with cohort-mode and CRAM cases; see docs/development/golden-cohort-gate.md. The branch was additionally put through an adversarial read-only review, which found the signed-frame merge blocker above — introduced by the #192 fix that made it reachable — before it could ship.

2.0.6¶

Unit coverage went from 25.68% to 70.57%, across 1,744 tests. The number was not the point: a mutation experiment had shown it overstated protection badly — confidence_assignment.py had 100% line coverage and a 21% mutation score, meaning its central condition could be inverted with the whole suite still green. That module now scores 57.9% raw / 84.6% adjusted at the same 100% line coverage.

No reported genotype or confidence label changes. That is measured, not asserted: the golden-cohort gate ran 122 pipeline runs before and after, and the Kestrel variant set, Confidence, Flag, adVNTR genotype and Flag are identical on every sample and every assembly. vntyper/scripts/kestrel_config.json is byte-identical to its pre-branch state. See docs/development/golden-cohort-gate.md.

⚠️ Required before deploying the image¶

• Rotate the two Redis passwords. They are in this repository's public history; removing them from source does not invalidate them.
• Set REDIS_PASSWORD. The web service now refuses to start without it.
• Deploy the worker before the API. run_vntyper_job gained an index_path parameter defaulting to None; old-API-to-new-worker is safe, new-API-to-old-worker is not.

Breaking changes (web service only; the CLI is unaffected)¶

• Alias-only cohort access no longer works — cohort_id and passphrase are both required. Cohorts with no stored passphrase hash are unopenable; treating a missing hash as "no passphrase" would be the bypass, and no cohort ever had a working hash (see below).
• /job-status/ returns a generic message for a failed job; the detail is logged server-side rather than returned.
• --output-name is now refused rather than silently ignored.

Web service hardening¶

docker/app — 1,679 LOC, internet-facing, processing patient-derived genomic data — had no tests, no lint, no type checking and no coverage. No automated check had ever read it. It is now formatted, linted, type-checked and 87% covered, and its path is in CI's filter so those gates fire.

• The passphrase check never worked. passlib 1.7.4's bcrypt backend probe is incompatible with bcrypt ≥ 4.1 and raised ValueError for every input, so no cohort could ever have been protected. Now on bcrypt directly.
• Compose set CELERY_BROKER_URL, which Celery prefers over the constructed URL, so the worker's real broker was the unauthenticated one. Removed, with a test blocking its return. Two different hardcoded Redis defaults were also committed, so with the variable unset the API and the worker authenticated differently against the same instance.
• Upload filenames are constrained by an allowlist before they build a path; uploads and request bodies are bounded; job ids are validated as UUIDs; cohort analysis no longer deletes its members' archives; /cohort-download/ cleans up its temporary archive.

Pipeline fixes¶

Each with a test written first and observed failing.

• vntyper report was dead — an uncaught TypeError.
• The Motif column was missing from every per-sample report; the pipeline emits Motif and the report keyed on Motifs, and unmatched columns are silently dropped.
• Every negative report was styled as positive, and screening states with no configured message fell through to "the screening was negative" — including Kestrel-positive-and-flagged.
• --extra-modules advntr,shark silently disabled both modules (exact-string membership, no comma split), so a typo yielded a silent Kestrel-only run.
• _construct_ncbi_accession compared against "hg19" while its only caller passes "GRCh37"/"GRCh38", so every non-chr1 GRCh37 NCBI name was built from the GRCh38 table.
• online_mode exited 0 on a failed job, so a wrapping subprocess.run(check=True) saw success.
• set -o pipefail added to three multi-stage pipes — without it a failing samtools sort upstream of samtools fastq still exits 0, genotyping a truncated FASTQ as if complete.
• The RU == 7 adVNTR flag now fires. It was added three hours and 45 minutes before the RU column existed, so it had never once evaluated. RU-7 calls move from positive to positive flagged; no row loses a flag and no genotype field changes.
• Jinja2 autoescaping on both the per-sample and cohort reports; shlex.quote on interpolated paths, regions and sample names.

Test infrastructure¶

• A patch-coverage gate at 80% of changed lines, which is what makes "touch a file, add tests for it" enforceable — demonstrated failing on an untested one-line change.
• A mutation-testing harness with the results and the equivalent-mutant classifications committed (docs/development/mutation-testing.md).
• A golden-cohort gate (docs/development/golden-cohort-gate.md).
• tests/builders.py domain-object builders, summary_steps.py for the five step-name literals four modules matched by exact string, and calibration.json recording the provenance of the ten calibrated constants.

Characterised, not fixed¶

Seventeen findings needing a domain decision or exceeding a hardening PR's scope were each pinned by a test and filed as #181–#197. Three worth reading: #186 (motifs_for_alt_gg ships empty, inert in the active branch but destructive in the uniform one), #181 (every (3n+1)-bp deletion discarded), and #185 (filter_final_dataframe fails open).

Closes #179.

2.0.5¶

Docker image now runs Python 3.12. Python 3.9 reached end of life on 2025-10-31. Every package on the numerical path is unchanged - bwa, samtools, fastp, bcftools, openjdk (Kestrel), pysam, numpy, pandas and biopython all resolve to the same versions - and the pipeline reproduces the expected genotype exactly. Only the reporting stack moved (matplotlib, plotly, igv-reports, jinja2). The package now requires Python 3.10+ and is tested on 3.10-3.13.

CI/CD pipeline rebuilt. The Docker image is split into a rarely-rebuilt base (conda environments, adVNTR, reference genomes) and a per-commit application image, so a commit now rebuilds in about 3 minutes instead of 40-70. Images are published only after their tests pass, and the build uses the repository's own source rather than cloning GitHub, so a pull request now tests the code in that pull request.

• Fixed pyproject.toml declaring numpy<2.0.0 while the conda environment installed numpy 2.0.2: pip install . inside the image downgraded conda's numpy, so the published image ran a different numerical stack than its own recipe declared.
• Fixed the TSV/CSV summary parsers silently truncating rows whose field count did not match the header; a malformed row is now logged and skipped without discarding the rest of the file.
• Recovered 30 unit tests that carried no unit marker and had therefore never run in CI, covering the Issue #136/#145 genotype tie-breaking logic.
• Added a fast image smoke tier that verifies every reference path config.json declares actually exists in the image.
• Added coverage reporting with a ratcheting floor and an 80% target.
• make check-all now runs in seconds and works on a fresh clone; it previously required a 1.1 GB download and a Docker daemon.

2.0.4¶

• Migrated all logging to per-module loggers (logging.getLogger(__name__)); log records now carry the emitting module name instead of root.
• Replaced deprecated datetime.utcnow() calls; emitted timestamps are unchanged.
• Ruff lint rules are now selected explicitly instead of extending ruff's defaults, so lint results no longer change when ruff widens its default rule set.
• mypy configuration moved from Makefile flags into pyproject.toml.
• Fixed .gitignore ignoring the tracked docs/ directory, which silently hid newly added documentation pages from Git.
• Fixed make test-quiet, which used a --log-cli=false flag pytest does not accept.
• Added AGENTS.md and CLAUDE.md with repository instructions for coding agents.

2.0.3¶

• Raised Kestrel max align and haplotype states to 40, fixing GDP inflation (#156).
• Handled pd.NA and None in flagging condition evaluation to prevent TypeErrors.
• Added a GDP inflation guard test with an anonymized hg38 sample.
• Added CITATION.cff with the Zenodo concept DOI.

2.0.2¶

• Removed specific motifs from exclude_motifs_right to prevent unwanted motif filtering.
• Clarified depth score conditions in confidence assignment (#154).
• Added a dynamic version variable to the docs via mkdocs-macros-plugin.
• Modernized docs deployment to use GitHub Actions Pages.
• Added a CFLAGS workaround for adVNTR compilation errors on GCC 14+.

2.0.1¶

• Disabled duplicate flagging in Kestrel configuration.
• Cleared motifs_for_alt_gg array to prevent unintended variant filtering.
• Fixed flagging to occur before variant selection, preventing selection of flagged variants (#145).
• Fixed confidence assignment to filter sub-threshold variants at the root rather than via override.
• Raised Low_Depth_Conserved_Motifs threshold from 0.2 to 0.4.
• Updated variant selection to prioritize depth score before haplotype count.
• Extracted frameshift deduplication into a helper function for clarity.
• Added comprehensive unit tests for the flagging module.

2.0.0¶

A complete rewrite and modernization of the VNtyper 2 pipeline.

• Modern Python packaging using pyproject.toml (PEP 517/518/621).
• Refactored Kestrel postprocessing with configurable thresholds via kestrel_config.json.
• HTML reports with embedded IGV integration for interactive variant inspection.
• Cohort analysis command with built-in pseudonymization for multi-sample studies.
• Docker multi-stage build for reproducible, lightweight container images.
• Comprehensive test suite with unit and integration tests, including downloadable test data.
• Multiple reference assemblies supported: hg19, hg38, GRCh37, and GRCh38.
• Modular architecture separating variant parsing, motif processing, scoring, flagging, and confidence assignment into individual modules.
• Snakemake workflow for scalable batch processing.

1.x¶

The original VNtyper release as described in Saei et al., iScience 2023. This version provided the initial alignment-free genotyping approach for MUC1 VNTR using Kestrel and adVNTR.