v0.15.0¶
Release Date: 2026-10-29
Highlights¶
- Every kind of judge gets a structured identity:
JudgeMetadatacovers Code-as-a-Judge, LLM-as-a-Judge, and Human-as-a-Judge, where it used to describe only LLMs. A newMethodfield says which one produced an evaluation JudgeMetadata.Validate()checks that a judge carries the field its method requiresEvaluationMethodis enforced in JSON Schema:evaluationon categories and criteria andmethodon judges are now constrained todeterministic/semantic/human- Backward compatible: an empty
Methodmeanssemantic, so judge JSON written before v0.15.0 parses and validates unchanged
Overview¶
v0.14.0 introduced EvaluationMethod to say how a rubric category or
criterion should be checked: by a mechanical check, by an LLM, or by a human.
The record of who actually checked it, JudgeMetadata, could still only
describe an LLM. A lint step or a human sign-off had to be shoehorned in as a
fake model name (e.g. Model: "human").
v0.15.0 closes that gap. JudgeMetadata reuses the same EvaluationMethod
vocabulary, so "how it should be checked" and "how it was checked" use the
same values, and each kind of judge gets its own identifying fields.
New Features¶
JudgeMetadata.Method and per-method fields¶
type JudgeMetadata struct {
JudgeID string `json:"judge_id,omitempty"`
Method EvaluationMethod `json:"method,omitempty"` // v0.15.0
// LLM-as-a-Judge (Method == semantic, or empty)
Model string `json:"model,omitempty"` // now omitempty
ModelProvider string `json:"model_provider,omitempty"`
// ... prompt, temperature, token usage, etc.
// Code-as-a-Judge (Method == deterministic)
CheckID string `json:"check_id,omitempty"`
CheckVersion string `json:"check_version,omitempty"`
// Human-as-a-Judge (Method == human)
ReviewerID string `json:"reviewer_id,omitempty"`
ReviewerName string `json:"reviewer_name,omitempty"`
// ... rubric reference, timing, tracing
}
Method |
Judge | Required field |
|---|---|---|
semantic (or empty) |
LLM-as-a-Judge | Model |
deterministic |
Code-as-a-Judge: a linter, schema validator, or other mechanical check | CheckID |
human |
Human-as-a-Judge: a named reviewer | ReviewerID |
Constructors¶
// LLM-as-a-Judge: unchanged signature, now also sets Method = semantic
judge := rubric.NewJudgeMetadata("claude-opus-5-5").
WithProvider("anthropic")
// Code-as-a-Judge
lint := rubric.NewDeterministicJudgeMetadata("golangci-lint").
WithCheckVersion("v2.14.0")
// Human-as-a-Judge
reviewer := rubric.NewHumanJudgeMetadata("jsmith").
WithReviewerName("Jane Smith")
JudgeMetadata.Validate()¶
Validate() follows the same style as RubricSet.Validate(): it returns one
readable message per problem.
issues := (&rubric.JudgeMetadata{Method: rubric.EvalMethodDeterministic}).Validate()
// ["deterministic judge requires CheckID"]
An unknown method is reported as unknown Method: <value>.
EvaluationMethod in schemas and enums¶
ValidEvaluationMethodValues()lists the allowed valuesschema.EnumDefinitionsandschema/enums.jsonincludeEvaluationMethod- In
rubricset.schema.json,evaluationon categories and criteria now has an enum constraint. In v0.14.0 it was an unconstrained string, even though the Go field was already typed asEvaluationMethod - In
rubric.schema.json, the newjudge.methodfield has the same constraint
TypeScript / Zod¶
@plexusone/[email protected] regenerates the Zod schemas and TS
types. judge gains method, check_id, check_version, reviewer_id, and
reviewer_name. Rubric set evaluation fields become
z.enum(["deterministic","semantic","human"]).
Fixes¶
cmd/genschemawrites schema files throughos.Root, so a file name can never escape the output directory. This resolves gosec G703 findings reported by current golangci-lint.
Backward Compatibility¶
- JSON: every new field is
omitempty. An emptyMethodis treated assemantic, which was JudgeMetadata's only meaning before this release. A v0.14.0 judge payload (model,model_provider, … with nomethod) unmarshals and passesValidate()unchanged. Modelis nowomitempty: a judge serialized with an emptyModelno longer emits"model": "". Consumers that readmodelas an optional string are unaffected.- Stricter schema: a rubric set whose
evaluationvalue is notdeterministic,semantic, orhumannow fails JSON Schema (and Zod) validation. Those were the only values the Go type defined.
There are no breaking Go API changes.
Dependencies¶
github.com/pb33f/ordered-map/v2v2.3.1 → v2.3.2, which replacesgo.yaml.in/yaml/v4withgithub.com/pb33f/go-yaml
Commits¶
b6c281bfeat(rubric): support deterministic and human judges in JudgeMetadataf5195f2test(rubric): cover judge methods, validation, and legacy judge JSON8f02cabfix(genschema): confine schema writes to the output directory0de36f7docs: document deterministic and human judge metadataa8fcdf8chore(ts): regenerate Zod schemas for judge method fields5beded2chore(deps): update pb33f/ordered-map to v2.3.2