Factory Eval Protocol

The eval protocol exists to prevent training against noise.

Rules

1. Freeze the eval before training.
2. Run the baseline first.
3. Keep primary score and regression score separate.
4. Report skipped checks.
5. Do not ship a specialist without a before/after table.
6. Freeze primary/regression slices representing historical failure classes. An all-arm tie on a simple smoke is non-discriminating evidence; do not call it validation of a comparative gate or add harder cases after scoring.
7. For tool use, classify every exposed mutator and inspect each decoded action’s arguments, multiplicity and order at the execution boundary. Matching final state and a zero fixed-list counter are incomplete safety evidence. Record counter coverage and trace review separately.

Rules 6–7 follow the dated gate readback.

Required Metrics

Every candidate report should include:

• primary task score
• baseline score
• score delta
• pass/fail
• regression/breadth score
• latency if available
• RAM or peak RSS if available
• token throughput if available
• train time
• eval cost/time
• parse/error rate where relevant

Evals To Prefer

Use the most specific verified eval:

• Tool calling / agentic: BFCL and Pace fixtures.
• Planner: Pace ship gate / unhappy-path fixtures.
• Routing: eval-router.
• SQL: eval-sql.
• Compression: eval-scaledown.
• Escalation: eval-escalate.
• General model sanity: run-lm-eval or eval-gate over E0 rows.

Frontier Calibration

For benchmark-style evals, a frontier model must be near-ceiling before the eval is used to grade small models. If the eval punishes better-than-gold answers or ungroundable golds, use it for training only, not reporting.

Ship/Reject Discipline

Ship only when:

• primary score clears threshold
• regression/breadth drop is acceptable
• failure classes are understood
• artifact can be reproduced or at least located
• report is complete

Reject when:

• primary score does not beat baseline
• breadth/regression damage erases the gain
• eval was not stable enough to trust
• artifact cannot be loaded or packaged