nw.validation#

Pluggable validation of finished work — the seam, not the checks.

What this is for. A render that finishes without raising is not a render that is right. A one-minute short shipped with a burnt-in lower third reading 1981 · Centr…; a ten-minute cut shipped two-thirds encoded with a container duration taken from its audio stream, so every duration check passed; a Ken Burns move jumped a whole pixel mid-panel for months. Each was found by a human watching the finished file, which is the most expensive detector we have and the only one that gets tired.

What this module owns, and what it deliberately does not. It owns the seam: how a check is declared, how the menu of available checks is assembled, how dependencies between them are resolved, in what order and with how much concurrency they run, and what a run of them reports. It owns no domain knowledge whatsoever — every actual check is a plugin, registered from wherever the knowledge lives (nw ships the two that nw.inspect already knew how to do; a check about on-screen type belongs next to tituli, one about camera motion next to burns).

Where it goes in a pipeline. Nowhere, by default. Validation is a thing you place, and where to place it is a judgement about cost and consequence:

before showing a result to a human, if the checks are cheap;
before an irreversible or outward-facing step — a publish, an upload, a send — where the checks earn their cost whatever they cost;
both, with a cheaper selection at the first point than the second.

So the surface is one function, validate(), and callers place it. Nothing in nw calls it for you; a gate you did not ask for is a gate that fires at the wrong moment.

Declaring a check:

from nw.validation import Check, checks

@checks.register_decorator("video.has_both_streams")
def _has_both_streams():
    return Check(
        name="video.has_both_streams",
        summary="the file actually contains a video and an audio stream",
        run=lambda target, ctx: (...),
        example_requests=("is the video ok", "did the render work"),
    )

example_requests is not decoration. A menu of forty checks is unusable by a human and unselectable by a model; the phrases a person actually says are what lets suggest() turn “make sure the captions look right” into a selection, which is how this reaches an MCP tool surface without a forty-item enum.

Examples

>>> report = validate("some.mp4", checks=())   # nothing selected, nothing run
>>> report.ok, list(report.findings)
(True, [])

Module Attributes

checks

The menu.

Functions

`register_check`([check])	Add a check to the menu, as a call or as a decorator.
`resolve_checks`(selection)	Selection plus everything it requires, with names resolved to Checks.
`plan_checks`(selection)	Order the selection into waves that may each run concurrently.
`validate`(target, *[, checks, max_workers, ...])	Run `checks` against `target` and report.
`suggest`(request, *[, include_paid])	Checks whose `example_requests` look like what the user just asked for.
`menu`(*[, cost])	Every registered check, name-ordered — what a user chooses from.

Classes

`Finding`(check, severity, message[, where, ...])	One thing a check noticed.
`CheckResult`(name[, findings, skipped, ...])	What one check produced, including the case where it could not run.
`ValidationReport`(target[, results, elapsed_s])	Everything a `validate()` run produced.
`Check`(name, summary, run[, requires, ...])	One validation, and everything a scheduler and a menu need to know.

Exceptions

ValidationError(report)

Raised by ValidationReport.raise_if_failed().

class nw.validation.Check(name, summary, run, requires=(), parallel_safe=True, cost='cheap', example_requests=(), requires_binaries=())[source]#

Bases: object

One validation, and everything a scheduler and a menu need to know.

name#: dotted and stable — it is what a user selects and what a requires refers to.

summary#: one line, for the menu.

run#: (target, context) -> findings. May also return a (findings, produced) pair when other checks depend on it.

requires#: names of checks that must run first, whose produced values arrive in context. Cycles raise at plan time.

parallel_safe#: whether it may run alongside its independent peers. False for anything that is not thread-safe or that saturates the machine on its own (a full decode).

cost#: rough wall-clock class — "free" (no subprocess), "cheap" (seconds), "dear" (a full pass over the media), or "paid" (spends money, e.g. a hosted OCR or transcription). "paid" is never selected by suggest(); it must be asked for by name.

example_requests#: things a person actually says that mean they want this check. What lets a menu of forty be navigated, and what an MCP surface matches against instead of exposing an enum.

requires_binaries#: external programs it shells out to. A missing one makes the check skip with a reason, never silently pass.

class nw.validation.CheckResult(name, findings=(), skipped='', error='', elapsed_s=0.0, produced=None)[source]#

Bases: object

What one check produced, including the case where it could not run.

skipped and error are kept distinct from “found nothing”, because conflating them is how a validation suite comes to report all-clear on a machine where half of it never ran. A missing binary is not a pass.

property ok: bool#: Ran, and found nothing at or above FAILING_SEVERITY.

class nw.validation.Finding(check, severity, message, where='', remedy=None, evidence=<factory>)[source]#

Bases: object

One thing a check noticed.

A finding is never a bare boolean. Whoever reads it — a human deciding whether to publish, or a model deciding what to fix — needs to know where in the work it is and what would make it go away, and a check that cannot say those two things has not finished its job.

check#: the name of the check that produced it.

severity#: "info", "warn" or "error"; only "error" makes a report not ok.

message#: what is wrong, in one sentence a human can act on.

where#: where in the work — a timestamp, a frame index, a shot id, a path. Free-form because the checks are, but never empty for anything above "info".

remedy#: what would fix it, when the check knows. None when it honestly does not.

evidence#: anything a reader would want to look at — an extracted frame’s path, the numbers behind the verdict.

exception nw.validation.ValidationError(report)[source]#

Bases: AssertionError

Raised by ValidationReport.raise_if_failed().

class nw.validation.ValidationReport(target, results=(), elapsed_s=0.0)[source]#

Bases: object

Everything a validate() run produced.

target#: what was validated.

results#: one per check that was selected, in the order they were run.

elapsed_s#: wall-clock for the whole run.

property ok: bool#

No failing findings and nothing that failed to run.

A check that errored is not a pass. Callers gating a publish on this get the conservative answer without having to remember to ask for it.

raise_if_failed()[source]#

Return self, or raise ValidationError — for a hard gate.

Return type:: ValidationReport

summary()[source]#

A few lines a human can read without unpacking the object.

Return type:: str

nw.validation.checks: Registry = <Registry nw.validation.checks>#: The menu. on_conflict="error" so a plugin that shadows a built-in fails loudly rather than quietly changing what “validated” means.

nw.validation.menu(*, cost=None)[source]#

Every registered check, name-ordered — what a user chooses from.

Return type:: tuple[Check, ...]

nw.validation.plan_checks(selection)[source]#

Order the selection into waves that may each run concurrently.

Every check in a wave has all its requirements satisfied by earlier waves, so the waves are the schedule: run each in turn, in parallel within it. A check that is not parallel_safe gets a wave to itself.

Raises:: ValueError – on a dependency cycle, or a requirement that is not registered — both at plan time, before anything has been spent.
Return type:: tuple[tuple[Check, …], …]

Examples

>>> plan_checks(())
()

Return type:: tuple[tuple[Check, ...], ...]

nw.validation.register_check(check=None, **kwargs)[source]#

Add a check to the menu, as a call or as a decorator.

As a call:

register_check(Check(name="video.duration", summary="...", run=...))

As a decorator on the run function, with the rest as keywords:

@register_check(name="video.duration", summary="...", cost="cheap")
def _duration(target, ctx): ...

nw.validation.resolve_checks(selection)[source]#

Selection plus everything it requires, with names resolved to Checks.

A user picks what they care about; what those checks need is not their problem. Raises KeyError for an unknown name — a silently dropped check is the failure this whole module exists to prevent.

Return type:: tuple[Check, ...]

nw.validation.suggest(request, *, include_paid=False)[source]#

Checks whose example_requests look like what the user just asked for.

Deliberately crude — a word-overlap score, not a model call — because this runs on every request and its job is to narrow forty items to a handful that a human or a model then confirms. "paid" checks are never suggested: money is asked for by name.

Return type:: tuple[Check, ...]

Examples

>>> suggest("")
()

nw.validation.validate(target, *, checks=(), max_workers=4, on_error='report')[source]#

Run checks against target and report.

Parameters:

target (Any) – whatever the checks understand — a path to a rendered file, a Project, a (video, annotations) pair. This module does not care; it is the checks that agree with their caller.
checks (Iterable[str | Check]) – names or Check objects. Requirements are pulled in automatically. Empty means empty: validation is placed, never assumed.
max_workers (int) – concurrency within a wave.
on_error (Literal['report', 'raise']) – "report" records a raising check as an errored CheckResult and carries on, so one broken plugin cannot hide the findings of the other nine. "raise" is for developing a check.

Return type:

ValidationReport

Returns:

A ValidationReport. Note that report.ok is False when a check errored, not only when one failed: a suite that could not run has not said the work is good.

Examples

>>> validate("x.mp4").ok
True