Skill evals

In paper console, every skill detail page has a Skill evals panel with three tabs: Evaluation, Improvements, and Eval spec. The spec is the rubric: one row per criterion, each with a weight and a kind. Run evaluation on the Evaluation tab checks the revision on screen against that rubric and the sessions the skill came from.

Skill evals

v7

EvaluationImprovementsEval spec

Latest · Manual · just now

Needs revision1 issue67%

Based on 3 sessions and evaluation criteria

Criteria 3/4 passed

• Names the exact command to runPassed
• States when not to use the skillPassed
• Does not paste the source transcriptFailed
• Handles a missing gatewayPassed

Re-run evaluation

Generate improvementNew

The result is a decision (Pass or Needs revision), a criteria score, and one row per criterion. Open a row for the requirement and the assessment. Criteria score = passed weight ÷ total weight. Important or essential failures require revision; session findings can also require revision.

After a run with failures, Generate improvement drafts a rewrite from the failed criteria. The panel widens into a review: the diff against the current SKILL.md, why the change was made, and the evaluation that prompted it. Accept as new revision saves it as a private revision and moves you onto it, so the next evaluation scores the rewrite rather than the text it replaced. Rejecting records the decision and changes nothing.

Evaluations are pinned to a revision. A new revision has no result until you run it; older runs stay under History.

A skill with no eval spec gets one drafted on the first run. Manage on the Eval spec tab edits it.

Open any skill to try it.