Skip to content

v0.15.0

Release Date: 2026-10-29

Highlights

  • Every kind of judge gets a structured identity: JudgeMetadata covers Code-as-a-Judge, LLM-as-a-Judge, and Human-as-a-Judge, where it used to describe only LLMs. A new Method field says which one produced an evaluation
  • JudgeMetadata.Validate() checks that a judge carries the field its method requires
  • EvaluationMethod is enforced in JSON Schema: evaluation on categories and criteria and method on judges are now constrained to deterministic / semantic / human
  • Backward compatible: an empty Method means semantic, so judge JSON written before v0.15.0 parses and validates unchanged

Overview

v0.14.0 introduced EvaluationMethod to say how a rubric category or criterion should be checked: by a mechanical check, by an LLM, or by a human. The record of who actually checked it, JudgeMetadata, could still only describe an LLM. A lint step or a human sign-off had to be shoehorned in as a fake model name (e.g. Model: "human").

v0.15.0 closes that gap. JudgeMetadata reuses the same EvaluationMethod vocabulary, so "how it should be checked" and "how it was checked" use the same values, and each kind of judge gets its own identifying fields.

New Features

JudgeMetadata.Method and per-method fields

type JudgeMetadata struct {
    JudgeID string           `json:"judge_id,omitempty"`
    Method  EvaluationMethod `json:"method,omitempty"` // v0.15.0

    // LLM-as-a-Judge (Method == semantic, or empty)
    Model         string `json:"model,omitempty"` // now omitempty
    ModelProvider string `json:"model_provider,omitempty"`
    // ... prompt, temperature, token usage, etc.

    // Code-as-a-Judge (Method == deterministic)
    CheckID      string `json:"check_id,omitempty"`
    CheckVersion string `json:"check_version,omitempty"`

    // Human-as-a-Judge (Method == human)
    ReviewerID   string `json:"reviewer_id,omitempty"`
    ReviewerName string `json:"reviewer_name,omitempty"`

    // ... rubric reference, timing, tracing
}
Method Judge Required field
semantic (or empty) LLM-as-a-Judge Model
deterministic Code-as-a-Judge: a linter, schema validator, or other mechanical check CheckID
human Human-as-a-Judge: a named reviewer ReviewerID

Constructors

// LLM-as-a-Judge: unchanged signature, now also sets Method = semantic
judge := rubric.NewJudgeMetadata("claude-opus-5-5").
    WithProvider("anthropic")

// Code-as-a-Judge
lint := rubric.NewDeterministicJudgeMetadata("golangci-lint").
    WithCheckVersion("v2.14.0")

// Human-as-a-Judge
reviewer := rubric.NewHumanJudgeMetadata("jsmith").
    WithReviewerName("Jane Smith")

JudgeMetadata.Validate()

Validate() follows the same style as RubricSet.Validate(): it returns one readable message per problem.

issues := (&rubric.JudgeMetadata{Method: rubric.EvalMethodDeterministic}).Validate()
// ["deterministic judge requires CheckID"]

An unknown method is reported as unknown Method: <value>.

EvaluationMethod in schemas and enums

  • ValidEvaluationMethodValues() lists the allowed values
  • schema.EnumDefinitions and schema/enums.json include EvaluationMethod
  • In rubricset.schema.json, evaluation on categories and criteria now has an enum constraint. In v0.14.0 it was an unconstrained string, even though the Go field was already typed as EvaluationMethod
  • In rubric.schema.json, the new judge.method field has the same constraint

TypeScript / Zod

@plexusone/[email protected] regenerates the Zod schemas and TS types. judge gains method, check_id, check_version, reviewer_id, and reviewer_name. Rubric set evaluation fields become z.enum(["deterministic","semantic","human"]).

Fixes

  • cmd/genschema writes schema files through os.Root, so a file name can never escape the output directory. This resolves gosec G703 findings reported by current golangci-lint.

Backward Compatibility

  • JSON: every new field is omitempty. An empty Method is treated as semantic, which was JudgeMetadata's only meaning before this release. A v0.14.0 judge payload (model, model_provider, … with no method) unmarshals and passes Validate() unchanged.
  • Model is now omitempty: a judge serialized with an empty Model no longer emits "model": "". Consumers that read model as an optional string are unaffected.
  • Stricter schema: a rubric set whose evaluation value is not deterministic, semantic, or human now fails JSON Schema (and Zod) validation. Those were the only values the Go type defined.

There are no breaking Go API changes.

Dependencies

  • github.com/pb33f/ordered-map/v2 v2.3.1 → v2.3.2, which replaces go.yaml.in/yaml/v4 with github.com/pb33f/go-yaml

Commits

  • b6c281b feat(rubric): support deterministic and human judges in JudgeMetadata
  • f5195f2 test(rubric): cover judge methods, validation, and legacy judge JSON
  • 8f02cab fix(genschema): confine schema writes to the output directory
  • 0de36f7 docs: document deterministic and human judge metadata
  • a8fcdf8 chore(ts): regenerate Zod schemas for judge method fields
  • 5beded2 chore(deps): update pb33f/ordered-map to v2.3.2