Evaluations Lab

What it does#

The loop is always the same: dataset, candidates, run, compare, promote. A dataset is what you test on, pasted text or a frozen snapshot of a real document's segments. A candidate is one setup under test, usually a workflow; to compare two models, build two one-step workflows differing only in the model.

A run scores each candidate against the dataset, with a cost estimate you confirm before anything starts. Comparing candidates against a baseline returns a verdict per candidate, improvement, regression, tradeoff, or tie, with the reasoning behind it, never a bare number; below a minimum sample size the result is honestly labelled "indicative only" instead. Promotion is the one deliberate way out of the Lab: it turns a winning candidate into a real production workflow, and only when the statistics genuinely support the win at your chosen margin.

Getting started#

1. Build a dataset: paste one segment per line, or snapshot a real document. Add a known-good translation per line to unlock the strongest metrics.
2. Define what you are comparing: create a candidate pointing at a workflow.
3. Launch a run: pick a dataset, candidates and scorers, then confirm the cost estimate.
4. Compare results against a baseline.
5. Promote the winner once the statistics support it.

eval dataset create --name "Support FAQ" --file ./lines.txt --target de-DE
eval candidate create --name "Qwen default" --workflow-id 4
eval run start --dataset 1 --candidate 4 --candidate 5 --budget 5.00 --yes
eval run compare --dataset 1 --baseline 4 --candidate 4 --candidate 5

Measuring the checker#

The Lab can measure a quality checker rather than a translation. Give it a gold set, the exact translations a person judged plus their error marks and an optional score, and a meta-evaluation reports whether a quality lens ranks the items the way the human did and whether it catches the errors they marked serious. The gold set is kept separate from the reference column and locked once labelled, so promoting a candidate can never quietly re-point the answer key.

Alongside the span-based gold set, a rules gold set gives each item a set of hand-written constraint rules a correct translation must satisfy. The rules verifier then translates each source and has a separate model, ideally a different family, judge every rule as pass or fail with a one-line reason; an example passes only if all its rules pass. It reports the example pass rate, the per-rule pass rate, the unreadable rate, the empty-candidate rate, and the cost per example, and can run the rules shown versus hidden to measure whether telling the model the rules helps.

One case is handled deliberately: an empty, whitespace-only, or invisible-only candidate, including a blank or held translation, fails every rule closed with the reason empty_candidate and is never sent to the judge. A blank translation cannot satisfy a rule, and a lenient same-family judge will happily pass one, so it is counted apart from a genuinely unreadable verdict and the judge call is skipped.

One arm of the rules verifier can run the Advanced Workflows committee itself against the gold rules, injecting them as the mission's verification rules. Each example is one advanced mission and run, and those rows are kept in the arm's project as a durable spend audit trail.

Works with#

• Advanced Workflows: the rules verifier can run the advanced committee as one arm, measured against a gold set of verification rules.
• Workflows: a candidate is normally just a pointer at a workflow, and promotion turns the winner into a production workflow.
• Documents: snapshotting a document copies its segments, and its current translations as references, into a frozen dataset.
• Engines: candidates and the LLM judge both resolve engines; a local-engine experiment costs nothing.
• Translation memory: a dataset snapshotted from a document inherits whatever translation memory already filled in, worth knowing before reading the scores.

Current limits#