← all projects

Dev Tools Active 2025

veriflow

ML/LLM verification harness

Source

The problem

Most ML release checklists live in someone's head or a wiki page - 'check the schema, check for drift, run the eval suite' - and get skipped under deadline pressure because nothing enforces them.

The approach

Built veriflow, a framework that encodes that checklist as executable gates defined in YAML: schema validation, distribution-drift detection against a reference dataset, regression tests against golden datasets, and eval-metric monitoring, runnable locally or wired into CI.

PythonML OpsData ValidationCLIYAML

veriflow turns “did we check the model before shipping it” from a checklist someone might skip into a set of gates that run automatically, locally or in CI.

the gap it fills

ML release checklists are usually informal: check the input schema hasn’t changed, check for data drift, rerun the eval suite, compare against a known-good baseline. Because none of that is enforced by tooling, it’s the first thing dropped when a release is late.

gates

veriflow defines each check as a gate, configured declaratively:

gates:
  - type: schema
    reference: schemas/input_v2.json

  - type: drift
    reference_dataset: data/reference_2025q1.parquet
    threshold: 0.05

  - type: regression
    golden_dataset: data/golden_set.parquet
    metric: accuracy
    min_score: 0.92

  - type: eval_monitor
    metrics: [precision, recall, f1]

Running veriflow check executes every configured gate against the current model/data and fails fast with a clear report of what broke.

local-first, CI-friendly

The same YAML config and CLI run identically on a laptop during development and inside a CI pipeline before merge - no separate “local mode” vs “CI mode” to keep in sync.

status

Active project: the core gate types (schema, drift, regression, eval monitoring) are implemented and usable via CLI or CI integration.

What I took away

  • A checklist only gets followed if skipping it is harder than running it - making the gates a single CLI command was the actual adoption unlock.
  • Distribution drift needs a concrete reference dataset and threshold, not a vague 'does this look different' check, or it never fires.
  • Golden-dataset regression tests catch the failure mode unit tests miss: the model still runs, it just quietly got worse.

0 comments

0/1000

Loading…