v0.9.0¶

Release Date: 2026-07-12

Highlights¶

IntegerScore Type: 1-5 scale scoring optimized for LLM judges
ReasonCode System: Standardized finding codes for automated repair workflows
CoverageReport: Generic coverage tracking via Extensions
v2 Schema Fields: Confidence, blocking codes, and domain extensions

Overview¶

v0.9.0 introduces a comprehensive v2 schema for LLM-as-Judge evaluations. The new IntegerScore type provides a 1-5 scale that research shows LLMs are most reliable at. The ReasonCode system enables automated repair workflows by categorizing findings with machine-readable codes. Coverage tracking is now generic and stored via the new Extensions field.

New Features¶

IntegerScore Type¶

A new 1-5 integer evaluation scale optimized for LLM judges:

import "github.com/plexusone/structured-evaluation/rubric"

// Score constants
rubric.ScoreUnacceptable  // 1
rubric.ScoreMajorRevisions // 2
rubric.ScoreAcceptable    // 3
rubric.ScoreGood          // 4
rubric.ScoreExcellent     // 5

// Helper methods
score := rubric.ScoreGood
score.String()        // "Good"
score.ToCategorical() // ScorePass
score.IsValid()       // true

ReasonCode System¶

Standardized finding identifiers for automated repair:

// Category prefixes
rubric.CategoryREQ    // Requirements
rubric.CategorySEC    // Security
rubric.CategoryARCH   // Architecture
rubric.CategoryDOC    // Documentation

// Pre-defined codes
rubric.CodeREQAmbiguous      // "REQ-AMBIGUOUS"
rubric.CodeSECMissingAuth    // "SEC-MISSING_AUTH"
rubric.CodeMETRICNoBaseline  // "METRIC-NO_BASELINE"

// Get repair prompt for automated fixes
finding := rubric.NewFindingWithCode("f1", "requirements", rubric.CodeREQAmbiguous, "Ambiguous", "...")
prompt := finding.GetRepairPrompt()

CoverageReport¶

Generic coverage tracking for spec completeness:

cr := rubric.NewCoverageReport()
cr.SetSection("components", 10, 8, []string{"card", "dialog"})
cr.SetSection("foundations", 4, 4, nil)
cr.ComputeOverall() // Simple average

// Weighted average
weights := map[string]float64{"components": 2.0, "foundations": 1.0}
cr.ComputeOverallWeighted(weights)

// Store in rubric via Extensions
r := rubric.NewRubric("dss-spec", "material-v3")
r.SetCoverage(cr)

// Retrieve later
coverage := r.GetCoverage()

Enhanced CategoryResult¶

Categories now support IntScore and Confidence:

result := rubric.NewCategoryResult("quality", rubric.ScorePass, "Good quality")
result.SetIntScore(rubric.ScoreGood)     // 1-5 scale
result.SetConfidence(0.85)               // 0.0-1.0
result.AddReasonCode(rubric.CodeREQAmbiguous)

// Check if human review needed
if result.HasLowConfidence() {
    // Route to human reviewer
}

Enhanced Finding¶

Findings now support reason codes and locations:

finding := rubric.NewFinding("f1", "requirements", rubric.SeverityHigh, "Missing auth", "No auth defined")
finding.SetCode(rubric.CodeSECMissingAuth)
finding.SetLocation("Section 3.2")
finding.SetRecommendation("Add OAuth2 authentication")

// Get automated repair prompt
prompt := finding.GetRepairPrompt()

Extensions Field¶

Store domain-specific metadata without schema changes:

r := rubric.NewRubric("custom", "doc.md")
r.SetExtension("coverage", coverageReport)
r.SetExtension("metrics", metricsData)

// Retrieve
if r.HasExtension("coverage") {
    coverage := r.GetCoverage() // Type-safe retrieval
}

MinIntScore Pass Criteria¶

Require minimum integer scores for approval:

r := rubric.NewRubric("eval", "doc.md")
r.PassCriteria.MinIntScore = rubric.ScoreGood // Require 4+

r.Evaluate(nil)
// Decision will fail if computed IntScore < 4

Use Cases¶

LLM-as-Judge Evaluations¶

Use IntegerScore for more reliable LLM evaluations:

// LLMs are more reliable with 5-level scales
result := rubric.NewCategoryResultWithIntScore(
    "completeness",
    rubric.ScoreGood,     // 4
    0.9,                   // High confidence
    "Meets requirements",
)

Automated Repair Workflows¶

Use ReasonCodes to drive automated fixes:

findings := report.Findings
for _, f := range findings {
    if f.Code != "" {
        prompt := f.GetRepairPrompt()
        // Send to LLM for automated fix
    }
}

Human Review Routing¶

Route low-confidence evaluations to humans:

if report.NeedsHumanReview() {
    // Send to human review queue
}

Migration Guide¶

Additive Changes¶

All changes in v0.9.0 are additive:

New types: IntegerScore, ReasonCode, CoverageSection, CoverageReport
New fields: IntScore, Confidence, ReasonCodes, Extensions, Blocking
New methods: SetCoverage(), GetCoverage(), SetExtension(), GetExtension()

Schema Version¶

v0.9.0 introduces schema versioning:

r := rubric.NewRubric("eval", "doc.md")
r.SchemaVersion // "v2"
r.IsV2()        // true

No Breaking Changes¶

No breaking changes in this release. Existing code continues to work unchanged.

Documentation¶

Added IntegerScore type with 1-5 scale scoring
Added ReasonCode system for categorized findings
Added CoverageReport for generic coverage tracking
Updated validation for v2 schema fields

Full Changelog¶

See CHANGELOG.md for the complete list of changes.